feat(mem_wal): support delete against non-nullable base columns

[hamersaw]

Problem

A tombstone carries the primary key and null in every other column, so ShardWriter::delete required every non-PK column to be nullable in the base table. That pushes a storage-engine detail into the user's schema for no reason the user can see.

Approach

Split the shard's schema in two:

• logical — the base table's schema, exactly as the caller declared it. The contract input is validated against, and the schema the scan path returns.
• storage — every non-PK top-level field widened to nullable. What the memtable, WAL entries, and SSTables physically carry.

This mirrors the logical/physical split SchemaAdapter already applies to JSON and view types in dataset/utils.rs, and rides the boundary that already exists — _tombstone is a physical column the SSTable schema carries and the base table does not.

Widening is top-level only. Arrow validates nullability just at the top level of a RecordBatch, so a null FixedSizeList/Struct needs no change below the top; a vector column's item field is untouched and gains no validity layer. Primary keys are never widened (Schema::unenforced_primary_key requires them non-nullable), so build_tombstone_batch still rejects a null, mistyped, or missing key — the delete path needed no new validation.

Where the contract is enforced

Ingress — put validates against the logical schema before the WAL append.

This is now the only gate. Both append (write/insert.rs) and merge_insert compare schemas with NullabilityComparison::Ignore, and the encoder derives validity from the array rather than the field, so a null that got past put would land in a non-nullable base column silently. Validating pre-append also matters for a second reason: a batch that is appended and only then rejected fails identically on every replay, leaving the shard unable to reopen — the same hazard that puts validate_index_configs ahead of claim_epoch.

WAL-only mode is covered as well; it previously validated nothing at all.

Egress — the scan narrows back to the logical schema after tombstones are filtered.

project_to_canonical documented that it emits its target_schema but did not: DataFusion derives ProjectionExec nullability from its expressions, not from the requested schema. A new SchemaRelabelExec makes that real. The same node widens arms so they agree before UnionExec/CoalesceFirstExec, both of which require exact schema equality — CoalesceFirstExec::new asserts it and would otherwise panic on the base-arm/WAL-arm nullability difference.

The narrowing doubles as a runtime assertion: if a tombstone ever escaped its filter, RecordBatch::try_new rejects the null instead of handing the caller a row of nulls.

Ordering falls out of this — carry_schema in the point-lookup path is built on the widened schema, because tombstones are still in flight until filter_tombstones_after_coalesce. vector_search and fts_search needed no changes; they already route every arm through project_to_canonical.

ensure_tombstone_column now always re-labels instead of passing through a batch that already has the column, so an entry written under an older storage schema replays into the current one.

Tests

13 new tests; 575 mem_wal lib tests pass.

• test_delete_against_non_nullable_base_column_round_trip — the headline: delete against a base table with a non-nullable non-PK column, survivors keep their values, and the scan reports the base table's own nullability.
• test_put_rejects_null_in_non_nullable_base_column / ..._wal_only_... — the ingress gate in both modes.
• test_build_tombstone_batch_nulls_non_nullable_base_column / ..._rejects_null_primary_key — widening works, PKs still strict.
• relax_* — top-level-only widening, nested fields untouched, _tombstone stays non-nullable, idempotence, metadata preserved (the PK marker rides on field metadata).
• schema_relabel::tests — widening, narrowing, narrowing rejects a surviving null, empty batches.

Not yet run

Draft because these are outstanding, not because the change is incomplete:

• Full cargo test -p lance --lib (~2700 tests) — only the mem_wal subset has been run.
• cargo clippy --all --tests --benches -- -D warnings.
• No end-to-end test yet confirming that Lance silently accepts a null into a non-nullable base column via merge_insert. The ingress gate is written as if it is load-bearing for base-table integrity, which is the safe assumption and what the code reading indicates, but it is unverified.

cargo fmt --all has been run.

🤖 Generated with Claude Code