feat(dataset): per-fragment column writes that survive compaction

[wkalt]

This adds FileFragment::write_column, which stages new data for one
fragment's column as a standalone data file and returns its
DataReplacementGroup without committing it. add_columns only appends a new
field; write_column may stage a file that answers for a field the fragment
already has, so a caller can recompute a column instead of only adding one.

It also extends DataReplacement to handle one more layout. Today
DataReplacement swaps a file when the field sets match exactly, or appends a
file when the fragment does not cover the fields at all, and rejects
everything else. That rejection covers the layout a long-lived column
actually reaches: compaction folds the column into a shared base file, no
file's field set matches a single-column replacement any more, and nothing
can replace the column again. Where the replaced fields all sit inside one
wider file, DataReplacement now tombstones them in place and appends the new
file to answer for them.

This supersedes #8207, which proposed a Dataset-level staging and commit
protocol for the same goal. Review there argued for building on the
fragment-level API instead.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The field-level replacement shape is appropriate, but the public staging boundary does not yet preserve schema, snapshot, or row-lineage contracts.

A viable revision should validate each staged recursive schema against the dataset, bind the staged read version to commit, and publish logical recomputations through a path that advances stable-row-ID update metadata.

[lance-gatekeeper]

write_column trusts a caller-provided Schema even when its field IDs already belong to different dataset types. That lets the staged file violate the manifest schema while Dataset::validate still succeeds; scans then decode bytes with the wrong logical type. Validate the full recursive schema against existing dataset fields before writing, and repeat that validation at commit.

Executed regression

I added reproduce_fragment_write_column_accepts_mismatched_existing_schema beside the existing fragment-write tests and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-main-target cargo test --locked -p lance reproduce_ -- --nocapture

The test creates a Float32 value field, stages Int32 [10, 20] under the same field ID, commits, calls validate(), and scans. Expected: staging or commit rejects the mismatched field schema. Observed: commit and validation succeed, and the scan returns Float32 values f32::from_bits(10) and f32::from_bits(20).

[lance-gatekeeper]

This metadata is written but never enforced, so a caller can present a newer commit read version and publish bytes prepared from an older snapshot. That permits a stale recomputation to overwrite a completed replacement. Make the staged result snapshot-bound: read and validate the recorded version and target backing state during commit, requiring recomputation after a relevant conflict.

Executed regression

I added reproduce_staged_read_version_is_not_enforced_at_commit and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-main-target cargo test --locked -p lance reproduce_ -- --nocapture

At version 1 the test stages stale values [30, 40] and winner values [50, 60]. It commits the winner as version 2, then commits the stale group while passing Some(2) to Dataset::commit. Expected: reject the stale file because its footer records version 1. Observed: the second commit succeeds and the scan returns [30, 40].

[lance-gatekeeper]

Returning only the fragment ID and data file discards the logical-update witness. Operation::DataReplacement does not advance stable-row-ID last_updated_at_version metadata, so delta reads can omit rows whose values this API changed. Route recomputation through the shared update/Merge machinery that records matched offsets, or extend this staged result and commit path to update row-lineage metadata atomically.

Executed regression

I added reproduce_fragment_write_column_does_not_advance_row_lineage and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-main-target cargo test --locked -p lance reproduce_ -- --nocapture

The test creates a version-1 dataset with stable row IDs, replaces both value rows, and commits version 2. Expected: _row_last_updated_at_version is [2, 2]. Observed: the values change, but the lineage column remains [1, 1].

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The primitive type mismatch is fixed and snapshot responsibility now matches Dataset::commit’s low-level contract. Two acceptance contracts remain: staged field identity can still diverge from the current manifest schema, and logical recomputation is still invisible to stable-row lineage.

A viable revision should validate complete recursive field identity at staging and against the commit-time manifest, then publish recomputations through a path that advances _row_last_updated_at_version.

[lance-gatekeeper]

data_type() does not establish that this is the same dataset field: it omits this field’s nullability and other Lance identity (recursive ID/path and logical metadata), and the comparison only uses the staging snapshot. As a result, nullable staged data is accepted for a non-null field, and a staged nullable file can also be committed after a concurrent Project tightens the manifest. Both produce a committed dataset that validate() accepts but scanning fails with unmasked nulls for non-nullable. Validate complete recursive field identity here and again against the commit-time manifest, or make schema projections conflict with this replacement.

Executed regressions

I added current-head cases beside the fragment write_column tests and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-main-target cargo test --locked -p lance reproduce_fragment_write_column -- --nocapture

One case created a non-null Int32 dataset field and staged nullable values containing a null. The other staged valid nullable data, concurrently projected that field to non-null, then committed the replacement. Expected: staging or commit rejects the incompatible field. Observed in both cases: commit and validate() succeed; scanning fails with unmasked nulls for non-nullable.

[lance-gatekeeper]

Returning only the fragment ID and data file discards the logical-update witness. Operation::DataReplacement does not advance stable-row-ID last_updated_at_version metadata, so delta reads can omit rows whose values this API changed. Route recomputation through the shared Update/Merge lineage machinery that records matched offsets, or extend this staged result and commit path to update row-lineage metadata atomically.

Executed regression

I added a current-head stable-row-ID case beside the fragment write_column tests and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-main-target cargo test --locked -p lance reproduce_fragment_write_column -- --nocapture

The test created a version-1 dataset with stable row IDs, replaced value from [1, 2] to [30, 40], and committed version 2. Expected: _row_last_updated_at_version becomes [2, 2]. Observed: the values change, but the lineage column remains [1, 1].

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The new field/nullability checks and symmetric Project conflict resolve the previous schema-race cases. Two correctness contracts remain: the staged schema still ignores storage-semantic layout metadata, and column recomputation still does not advance stable-row lineage.

A viable revision should derive or validate the writer schema against every layout-defining field attribute, then carry affected row offsets through the replacement commit so _row_last_updated_at_version advances atomically.

[lance-gatekeeper]

Default::default() leaves compare_metadata false. That is unsafe at this file-writing boundary: lance-encoding:packed is field metadata, but it changes physical field coverage. Removing that marker while preserving names, IDs, types, nullability, and children is accepted; the writer then advertises child fields [1, 2] instead of the packed parent [0], so DataReplacement can classify the file as disjoint/all-null coverage rather than replacing the packed field. Validate every layout-defining attribute here, or derive the writer schema from the matched dataset field.

Executed regression

I added reproduce_fragment_write_column_rejects_packed_metadata_mismatch on this head and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-verify-531-target cargo test --locked -p lance reproduce_fragment_write_column_rejects_packed_metadata_mismatch -- --nocapture

The test stages an otherwise-identical V2.1 packed struct after removing only lance-encoding:packed. Expected: write_column rejects the schema. Observed: the assertion result.is_err() fails because it returns Ok(DataReplacementGroup(... DataFile { fields: [1, 2], column_indices: [0, 1], ... })).

[lance-gatekeeper]

Returning only the fragment ID and data file still discards the logical-update witness. The DataReplacement manifest path changes files and indices but never advances stable-row-ID last_updated_at_version, so delta reads can omit rows whose values this API recomputed. Route this through the shared Update/Merge lineage machinery, or carry affected offsets in the staged result and update lineage atomically at commit.

Executed regression

I added reproduce_fragment_write_column_advances_row_lineage on this head and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-verify-531-target cargo test --locked -p lance reproduce_fragment_write_column_advances_row_lineage -- --nocapture

The test creates a version-1 dataset with stable row IDs, writes [30, 40] through write_column, and commits DataReplacement as version 2. Expected: _row_last_updated_at_version is [2, 2]. Observed: the assertion fails because it remains [1, 1].

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The staging checks now cover type, nullability, and row count, but three correctness contracts remain: layout-defining metadata is ignored, schema-changing Project can rebase across a staged replacement, and logical recomputation does not advance stable-row lineage.

A viable revision should normalize the writer schema to the manifest’s physical layout, conflict schema evolution with staged replacements, and carry affected row offsets into atomic lineage updates.

[lance-gatekeeper]

Default::default() leaves compare_metadata false. That is unsafe at this file-writing boundary: lance-encoding:packed is metadata, but it changes physical field coverage. Removing only that marker from an otherwise identical packed struct is accepted, and the staged DataFile advertises child coverage instead of the packed parent, so DataReplacement can classify it as disjoint/all-null coverage rather than replacing the field. Validate every layout-defining attribute here, or derive the writer schema from the matched dataset field.

Executed regression

I added reproduce_fragment_write_column_rejects_packed_metadata_mismatch on this head and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-delta-60690216-target cargo test --locked -p lance reproduce_fragment_write_column_rejects_packed_metadata_mismatch -- --nocapture

Expected: write_column rejects the V2.1 schema after only lance-encoding:packed is removed. Observed: the assertion fails because it returns Ok(DataReplacementGroup(... DataFile { fields: [1], column_indices: [0], ... })).

[lance-gatekeeper]

This staged result has no protection from concurrent schema evolution, while the conflict resolver treats Project and DataReplacement as compatible in both directions. Even with the correct read version, a nullable file can therefore commit after Project makes the manifest field non-nullable, leaving a dataset that cannot be scanned. Make these operations conflict symmetrically, or revalidate the staged field against the commit-time manifest.

Executed regression

I added reproduce_data_replacement_conflicts_with_projected_nullability on this head and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-delta-60690216-target cargo test --locked -p lance reproduce_data_replacement_conflicts_with_projected_nullability -- --nocapture

The test stages nullable data at version 1, commits a valid nullability-tightening Project as version 2, then commits the replacement using read version 1. Expected: Error::RetryableCommitConflict. Observed: version 3 commits, then scanning fails with Found unmasked nulls for non-nullable StructArray field "value".

[lance-gatekeeper]

Returning only the fragment ID and data file discards the logical-update witness. The DataReplacement manifest path changes files and indices but never advances stable-row-ID last_updated_at_version, so delta reads can omit rows whose values this API recomputed. Route this through the shared Update/Merge lineage machinery, or carry affected offsets in the staged result and update lineage atomically at commit.

Executed regression

I added reproduce_fragment_write_column_advances_row_lineage on this head and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-delta-60690216-target cargo test --locked -p lance reproduce_fragment_write_column_advances_row_lineage -- --nocapture

The test creates a version-1 dataset with stable row IDs, writes [30, 40] through write_column, and commits DataReplacement as version 2. Expected: _row_last_updated_at_version is [2, 2]. Observed: the assertion fails because it remains [1, 1].

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The new row-lineage coverage is sound, but two schema-safety contracts still fail: layout-defining field metadata is ignored at staging, and schema-changing Project commits remain compatible with staged DataReplacement.

A viable revision should derive or validate the staged writer schema against the manifest’s physical layout and either conflict Project/DataReplacement symmetrically or revalidate staged fields against the commit-time manifest.

[lance-gatekeeper]

Default::default() leaves compare_metadata false, so this check accepts a schema after the layout-defining lance-encoding:packed marker is removed. The writer then derives different physical field coverage from an otherwise identical field, so a replacement can be classified against the wrong layout. Derive the writer schema from the matched manifest field, or validate every layout-defining attribute before writing.

Executed regression

On exact head e21601bd4f4bd925b46d93f964678c62fbd30137 I added gate_repro_write_column_rejects_packed_metadata_mismatch beside the existing fragment-write tests and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-main-target cargo test -p lance gate_repro_write_column_rejects_packed_metadata_mismatch -- --nocapture

The test creates a packed struct, removes only lance-encoding:packed while preserving IDs, types, children, and nullability, then expects write_column to reject it. Observed: the command exits 101 because write_column returns Ok and the rejection assertion fails.

[lance-gatekeeper]

This staged result is not bound to the commit-time schema, while the conflict resolver treats Project and DataReplacement as compatible in both directions. A nullable file can therefore commit after Project makes the manifest field non-nullable, leaving a committed dataset that cannot be scanned. Make these operations conflict symmetrically, or revalidate the staged field against the final manifest during commit.

Executed regression

On exact head e21601bd4f4bd925b46d93f964678c62fbd30137 I added gate_repro_data_replacement_conflicts_with_nullability_project and ran:

CARGO_TARGET_DIR=/home/agent/tmp/gate8313-main-target cargo test -p lance gate_repro_data_replacement_conflicts_with_nullability_project -- --nocapture

The test stages [10, null] at version 1, commits a valid nullability-tightening Project at version 2, then commits the replacement using read version 1. Expected: a retryable conflict. Observed: the replacement commits, and scanning fails with Found unmasked nulls for non-nullable StructArray field "value".