[austin-barrington]

Data corruption, in 0.9.2-beta and earlier. When a measurement's first flush contains writes with different fields, values are stored in the wrong column. For example, m,host=a used=1 then m,host=a free=2, within one flush interval, stores the 2 under used and leaves free null. Series with different tags swap tag values the same way. It reproduces 3/3 on a single node and on a sharded cluster. The SQL insert path (HYPERBYTEDB_DISABLE_ARROW_INSERT=1) stores it correctly.

Cause. A flush merges a measurement's prepared Arrow batches with concat_batches under the first batch's schema, which stacks columns by position. […, used] and […, free] have the same column count and type, so free's values went under used. The comment said it only stacks batches with identical schemas, but nothing made them identical. Once the table exists, later writes are usually prepared with every column, which is why this mostly hits a measurement's first flush.

Fix.

• Batches are stacked by name. A column a batch lacks is filled with null for a field, or "" for a tag, which is what the series table stores for an absent tag. A type that differs widens as ingest does: float over integer, unsigned over signed, string over dictionary.
• The prepared series batch is padded to the table's tags before insert, as fact batches already were. Tags the table plan doesn't list yet are kept: on the prepared path that plan comes from metadata only.

Tests.

• Unit: fact batches with different fields, a field whose type differs, and series with different tags; padding of the series batch, including a tag missing from the plan.
• End to end, real chDB: three writes to a new measurement through the prepared path in one flush land in their own columns. It fails on the old merge.
• Real server: the original single-node reproduction now stores every value correctly.
• CI-equivalent run (libchdb 26.9.0, -D warnings): all green. A first version of the padding dropped tags the plan didn't list. The sharding suite caught that, and this version keeps them.

Found while testing the partial-write fix (#122); independent of it.

[cursor]

PR Summary

High Risk
Corrects silent on-disk corruption on the default Arrow flush path for new measurements and mixed-schema batches; changes merge/insert semantics for prepared WAL coalesce and series inserts.

Overview
Fixes data corruption on the prepared Arrow flush path (default in 0.9.2-beta and earlier): when several WAL batches for the same measurement were merged in one flush with different field or tag columns, concat_batches stacked columns by position, so values (e.g. free) could be stored under the wrong column (e.g. used). The SQL insert path was unaffected.

Merge behavior in arrow_coalesce now uses concat_by_name: union columns by name, fill missing fields with null and missing tags with "", and widen conflicting types (float/int, dictionary/string) like ingest does. Series batch merges use the same helper before dictionary normalization.

Insert alignment: prepared series batches are pad_series_batch_to_ensured before chDB insert (matching fact padding)—table tag order/types, "" for absent tags, and extra tag columns kept when metadata lags.

Unit tests cover name-based merge, type widening, and series padding; a compat test flushes mixed-field/tag line protocol and asserts query results.

^{Reviewed by Cursor Bugbot for commit e6b2df8. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e6b2df8. Configure here.}

[cursor]

Unsigned and float widen incorrectly

Medium Severity

wider_type promotes Float64 and UInt64 to Float64, but ingest widening via widen_field_disc treats those types as incompatible and keeps the stored type. A flush that stacks a uint batch with a float batch can then disagree with the metadata/table type. pad_record_batch_to_ensured clones the widened column without casting, so RecordBatch::try_new fails and the flush errors, or unsigned values are stored as floats when ingest would have nulled them.

 

^{Reviewed by Cursor Bugbot for commit e6b2df8. Configure here.}

[austin-barrington]

Fixed in b20c5da. wider_type now widens only the pairs ingest's widen_field_disc widens: float over signed integer, and unsigned over signed. Float and unsigned keep the first type. Also, pad_record_batch_to_ensured now casts every field column to the table's type, which is the authority, with values that can't convert becoming null. A column of a different type used to fail RecordBatch::try_new, and the flush retried it forever. That gap predates this PR, but it's the same path. Unit tests: float_and_unsigned_keep_the_first_type and a_prepared_column_is_cast_to_the_tables_type. The compat and sharding suites pass.

[cursor]

Stale comment

Left a non-blocking comment; this PR is not approved. Cursor Bugbot reported an unresolved medium-severity finding, and that thread is still open after the follow-up commit. Human review is needed, and no reviewers were assigned.

  

_{Sent by Cursor Approval Agent: Pull Request Router and Approver}

[cursor]

Left a non-blocking comment; this PR is not approved. Cursor Bugbot’s medium-severity finding is still an open review thread, and Bugbot has not re-reviewed the current head. Human review is needed, and no reviewers were assigned.

  

_{Sent by Cursor Approval Agent: Pull Request Router and Approver}