lancedb 0.27.2 → 0.37.1 + arrow-schema 57 → 59: real storage-engine migration, not a bump (#417, #414)

[explosivebit]

Scoped while assessing v0.37.0. Not taken into that release — separated deliberately, not dropped.

Why these two travel together

lancedb 0.37.1 pulls arrow-schema 58.4.0. Our code targets a different arrow-schema line, so bumping either PR alone breaks:

error[E0308]: mismatched types
  --> crates/forgeplan-core/src/db/migrate.rs:50:55
   | expected `lancedb::arrow::arrow_schema::Schema`, found `arrow_schema::Schema`

#414 (arrow-schema 57→59) and #417 (lancedb 0.27.2→0.37.1) must move as one unit.

What actually breaks

Confirmed by CI on both PRs (Check, Lint & Format), crates/forgeplan-core/src/db/migrate.rs, 637 lines:

• add_columns changed signature — AddColumnsBuilder takes 0 arguments now, we pass 2 (NewColumnTransform + Option<_>).
• AddColumnsBuilder is no longer a Future — the .await on it no longer type-checks.
• The type mismatch above, from the arrow-schema version skew.

Four call sites in migrate.rs need rewriting against the new builder API, not just a version bump.

Why not in v0.37.0

That release already carries a breaking change to R_eff scoring (PRD-086). Landing a storage-engine migration through ten minor lancedb versions in the same release means any post-release regression has two independent, unrelated causes to rule out instead of one. The risk here is in on-disk behaviour, not diff size — migrate.rs is the schema-migration path, not incidental plumbing.

Scope for whoever picks this up

• Rewrite the 4 add_columns call sites in migrate.rs against 0.37.1's builder API.
• Re-run the migration test suite against a workspace carrying the OLD on-disk schema (not just a fresh init) — this is a migration path, the interesting case is upgrading data that already exists.
• Check whether lancedb 0.37.1 changed anything else touching LanceStore beyond add_columns — 10 minor versions is a wide surface; this issue only names what CI caught, not a full changelog read.

Refs: #414, #417