[Bug] Ordered compaction loses logical commit TSO values in linked DUP rowsets

[HappenLee]

Search before asking

Searched existing issues for commit TSO and ordered/link compaction. Related reader and pruning reports are #67994 and #67995. This report concerns loss of the per-segment TSO interpretation during compaction, which is a separate problem.

Version

Apache Doris master, source baseline 84ebcf71bb52735c22034d9185c51e373caa9acd.

This is a source-derived correctness report. The end-to-end reproduction described below has not been executed for this report.

What's Wrong?

Ordered/link compaction can turn several single-version rowsets of a local DUPLICATE KEY table with ROW binlog enabled into a multi-version rowset without materializing their logical commit TSO values.

For a published single-version rowset, __DORIS_COMMIT_TSO_COL__ contains a physical placeholder of 0. Its logical value comes from that rowset's commit TSO metadata. Ordered compaction links the existing segment files, combines the version interval, and records an overall commit TSO range. It does not preserve which source TSO belongs to each linked segment.

The reader substitutes a constant TSO only for a single-version rowset. Once the output spans multiple versions, it reads the physical column instead. The linked files still contain 0.

For example, using illustrative metadata values:

Before compaction:
  Rowset A: version=[7,7], commit_tso=[1000,1000], physical TSO=0
  Rowset B: version=[8,8], commit_tso=[2000,2000], physical TSO=0
  Logical row values: 1000 for A, 2000 for B

After linking the files:
  Output: version=[7,8], commit_tso=[1000,2000]
  Physical row values are still 0
  The single-version substitution no longer applies

Consequences inferred from these code paths:

• Projecting the hidden column can return 0 instead of the original commit TSO.
• A commit_tso > 0 predicate can incorrectly reject matching rows.
• A commit_tso <= 1500 predicate can incorrectly accept rows from B, including when placeholder ZoneMaps make the predicate appear always true.
• Historical reads relying on these boundaries can return an incorrect row set.

Disabling ZoneMap pruning alone cannot restore the missing logical values.

The scope is the local ordinary DUPLICATE KEY base-table ordered-compaction path. A base table with ROW binlog enabled is not the dedicated row-binlog tablet. The examined admission path rejects Cloud mode and ordinary merge-on-write tablets; this report does not claim those paths are affected by this specific mechanism.

What You Expected?

Compaction must preserve the logical commit TSO of every row. In the example, commit_tso > 0 must retain both input batches, while commit_tso <= 1500 must retain only A's rows. Projection, filtering, and historical reads must agree before and after compaction.

How to Reproduce?

The following is a proposed BE fixture or controlled-cluster validation procedure, not a claimed successful SQL reproduction:

1. Create a local DUPLICATE KEY table with ROW binlog enabled so its schema includes __DORIS_COMMIT_TSO_COL__.
2. Produce two published single-version rowsets with distinct positive commit TSOs and record the expected TSO for each row from the source rowset metadata.
3. Ensure the normal ordered-compaction admission conditions hold: matching schema versions, strictly ordered non-overlapping key ranges, sufficiently large segments, and no other conditions forcing a rewrite. Enable enable_ordered_data_compaction.
4. Trigger cumulative or base compaction. Verify through instrumentation or logs that handle_ordered_data_compaction() actually links the target input segments. Merely completing compaction is insufficient to exercise the bug.
5. Open fresh readers on the output rowset and compare projected commit TSOs and the two boundary predicates above with the saved oracle. Compare the actual row identities, not only row counts.
6. Repeat with ordinary rewrite compaction as a control, then validate equivalent historical-read boundaries.

Anything Else?

Source references at the examined baseline:

• Ordered-compaction admission.
• Linking segment files and building the output metadata.
• Single-version commit TSO substitution.

A minimal fix direction is to exclude ordinary base-table rowsets that rely on commit TSO placeholders from ordered/link compaction and use the normal rewrite path. Retaining the linking optimization would require preserving the source-segment-to-TSO mapping and teaching readers to use it, or otherwise materializing the real values. An overall minimum/maximum TSO range is insufficient.

Related PR: #68097 resolves logical commit TSO readers when the source metadata is available. Its single-version rule cannot reconstruct the lost mapping in this multi-version output, and it does not change the ordered-compaction path. This failure mode predates that PR.

[Doris-Breakwater]

Initial assessment

Static triage: confirmed correctness defect at 84ebcf71bb52735c22034d9185c51e373caa9acd; recommended priority: high. The report has not demonstrated the failure in a running cluster, so the runtime manifestation is still unverified, but the loss mechanism is established by the code and is sufficient to treat this as a real bug rather than a speculative hypothesis.

The issue is currently open with no labels, assignee, milestone, or linked development work. It should receive the project's bug/correctness and BE storage/compaction labels.

Verified facts

1. The hidden commit-TSO column is defined with physical default 0 (ColumnDefinition.java). Publish assigns the real TSO to rowset metadata (Rowset::make_visible).
2. BetaRowsetReader passes the enclosing rowset's version and commit-TSO range to every segment. Segment::new_column_iterator substitutes a logical constant only when that rowset version is a singleton (version.first == version.second); otherwise it uses the physical column (reader setup, substitution condition).
3. Ordered compaction calls link_files_to() for each input, builds one spanning-version output, and stores only commit_tso_range(_input_rowsets) in its metadata. It does not persist an input-rowset/segment-to-TSO mapping (ordered output construction).
4. The admission code rejects Cloud mode and ordinary unique-key MoW tablets, but it has no exclusion for an ordinary DUP base tablet whose schema has commit_tso_col_idx() != -1 (admission path). Ordered compaction is enabled by default; the default minimum segment size is 10 MiB.

Therefore, after linking singleton rowsets with different commit TSOs into a multi-version output, the reader supplies the output version/range to each linked segment, fails the singleton check, and reads the retained physical 0. The stated consequences follow:

• projection can expose 0;
• commit_tso > 0 can prune/reject affected rows;
• commit_tso <= target can admit rows whose source commit TSO is newer than target;
• disabling ZoneMap pruning cannot repair row-level values because the physical value itself is still 0.

The normal rewrite path is a valid control: each singleton input is read with its own metadata-backed constant, and those logical values are then written into the multi-version output. A single output min/max range cannot reconstruct the per-segment values after files have been linked.

This also matches the discussion on #68097: the ordered-link case was independently acknowledged there as a pre-existing correctness issue and explicitly deferred to separate producer-side work. #68097's reader changes do not preserve the mapping lost here.

What is still missing

No end-to-end or BE fixture has run the exact path yet. Runtime confirmation should capture:

• the exact BE commit and the two relevant configuration values;
• input rowset versions and singleton commit TSOs, then output version and TSO range;
• the start to do ordered data compaction log or compaction-profile evidence proving the link path ran;
• row identity plus projected commit TSO before and after compaction, and results for both commit_tso > 0 and a boundary between the two source TSOs.

These are needed to validate the user-visible manifestation, not to establish the source-level root cause. Logs unrelated to ordered-path selection would add little value.

Recommended next steps

1. Apply the minimal safe fix: make ordinary base-table rowsets with commit_tso_col_idx() != -1 ineligible for ordered/link compaction, while leaving the dedicated row-binlog path distinct. Returning false from ordered admission lets normal rewrite compaction materialize the logical values.
2. Add a focused case to be/test/storage/compaction/ordered_data_compaction_test.cpp: create a TSO-enabled DUP schema, two non-overlapping singleton rowsets with physical 0 and different positive metadata TSOs, make the ordered size threshold deterministic, and verify the ordered path is rejected after the fix.
3. In the same fixture or a controlled regression, run normal rewrite compaction and assert row identities and TSO projection/predicates before and after. The existing time_travel_p0/test_time_travel_dup test is insufficient because it does not prove which compaction path ran.
4. If retaining the linking optimization is desired later, persist a durable source-segment-to-TSO mapping and teach every relevant reader/pruning path to consume it. This is a larger storage-format/metadata design; the overall TSO range is not enough.

No code change was made as part of this triage.

Breakwater-GitHub-Analysis-Slot: slot_ddd73f540137