Skip to content

[feat](row binlog):support cloud binlog and independent binlog tablet#65810

Open
Userwhite wants to merge 8 commits into
apache:masterfrom
Userwhite:support_cloud_binlog
Open

[feat](row binlog):support cloud binlog and independent binlog tablet#65810
Userwhite wants to merge 8 commits into
apache:masterfrom
Userwhite:support_cloud_binlog

Conversation

@Userwhite

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

This PR implements Row Binlog based on an independent row-binlog tablet
model for both non-cloud mode and cloud mode.

Before this change, Row Binlog data was tightly coupled with the base
tablet write/compaction path, and cloud mode did not have a complete end-
to-end implementation. This made it difficult to keep Row Binlog behavior
consistent across storage-compute integrated and storage-compute
separated deployments, especially for MOW tables, historical values,
partial update/upsert publish conflicts, and compaction.

The new implementation introduces row-binlog companion tablets and wires
them through FE metadata, BE write/publish/compaction/query paths, and
Cloud Meta Service. In non-cloud mode, Row Binlog write and compaction
are refactored around the independent tablet model. In cloud mode, the
same model is extended with cloud-specific rowset commit, publish,
delete-bitmap, metadata, and compaction handling.

Main changes:

  1. Independent Row Binlog tablet model
  • Store Row Binlog data in companion row-binlog tablets instead of
    coupling it directly with base tablet rowsets.
  • Track the relationship between base tablets and row-binlog tablets
    through aligned tablet ids.
  • Represent Row Binlog indexes with dedicated materialized-index states.
  • Keep Row Binlog schema aligned with master: __DORIS_BINLOG_TSO__,
    __DORIS_BINLOG_LSN__, __DORIS_BINLOG_OP__.
  1. Non-cloud write, publish, and compaction path
  • Support writing Row Binlog data through independent tablets in non-
    cloud mode.
  • Propagate Row Binlog tablet information through FE sink planning and BE
    load/write context.
  • Refactor Row Binlog compaction to reuse cumulative-compaction style
    logic.
  • Add CumulativeCompactionBinlogPolicy for non-cloud Row Binlog
    compaction.
  • Preserve MOW historical-value behavior, partial update/upsert
    semantics, and delete-bitmap correctness.
  1. Cloud write, publish, and metadata path
  • Add cloud Row Binlog companion tablet creation and metadata flow.
  • Add paired base-rowset + row-binlog-rowset commit through Meta Service.
  • Support Cloud publish conflict handling for partial update and upsert.
  • Make Cloud delete-bitmap cache, rowset builder/writer, tablet manager,
    and transaction paths aware of Row Binlog rowsets.
  • Persist Row Binlog tablet statistics and compaction metadata in Cloud
    Meta Service.
  1. Cloud Row Binlog compaction
  • Add CloudBinlogCumulativeCompactionPolicy.
  • Keep binlog-specific candidate selection separate from normal
    compaction paths.
  • Add filter_new_visible_rowsets to avoid compacting newly visible
    rowsets too early unless backlog requires it.
  • Fix cloud candidate selection so cloud rowsets are not incorrectly
    filtered as non-local rowsets.
  1. Query, schema change, restore, and observability
  • Update Row Binlog scan/read paths to use TSO/LSN correctly.
  • Support Row Binlog behavior across schema change, restore, tablet
    report, SHOW/proc paths, and compaction actions.
  • Expose Row Binlog tablet/index information where needed for debugging
    and operation.
  1. Tests
  • Add/update BE unit tests for non-cloud and cloud Row Binlog write,
    rowset builder/writer, transaction, and compaction policy behavior.
  • Add/update regression tests for Row Binlog basic behavior, compaction,
    schema change, multi-segment reads, Cloud compaction, and Cloud publish
    conflict.
  • Update regression ordering to use __DORIS_BINLOG_TSO__, __DORIS_BINLOG_LSN__ where result order depends on commit/visible TSO
    semantics.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Userwhite Userwhite changed the title Support cloud binlog [feat](row binlog):support cloud binlog and independent binlog tablet Jul 20, 2026
@Userwhite
Userwhite force-pushed the support_cloud_binlog branch from 185bbfe to a42d600 Compare July 20, 2026 11:52
@Userwhite

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 29.93% (91/304) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.52% (1910/2464)
Line Coverage 64.26% (34176/53181)
Region Coverage 64.59% (17593/27238)
Branch Coverage 53.78% (9435/17544)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29573 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a42d60026150be6fa921f64e362fdb388bc33e58, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17695	4084	4042	4042
q2	2084	343	198	198
q3	10531	1423	843	843
q4	4755	480	340	340
q5	8366	873	594	594
q6	333	172	136	136
q7	850	846	609	609
q8	10630	1563	1535	1535
q9	5869	4328	4331	4328
q10	6821	1747	1450	1450
q11	510	362	316	316
q12	734	575	465	465
q13	18125	3362	2785	2785
q14	271	258	242	242
q15	q16	792	776	707	707
q17	978	1084	933	933
q18	6804	5755	5587	5587
q19	1217	1237	1117	1117
q20	779	664	600	600
q21	5519	2611	2455	2455
q22	423	356	291	291
Total cold run time: 104086 ms
Total hot run time: 29573 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4398	4311	4293	4293
q2	279	317	210	210
q3	4585	4969	4356	4356
q4	2063	2154	1339	1339
q5	4406	4234	4253	4234
q6	227	172	126	126
q7	2216	1963	1733	1733
q8	2482	2188	2197	2188
q9	8146	7845	7848	7845
q10	4702	4684	4250	4250
q11	560	418	374	374
q12	753	761	539	539
q13	3216	3574	2872	2872
q14	291	297	282	282
q15	q16	705	738	655	655
q17	1372	1316	1479	1316
q18	7918	7452	7388	7388
q19	1186	1070	1103	1070
q20	2203	2197	1938	1938
q21	5231	4582	4375	4375
q22	510	446	410	410
Total cold run time: 57449 ms
Total hot run time: 51793 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 176800 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a42d60026150be6fa921f64e362fdb388bc33e58, data reload: false

query5	4341	616	477	477
query6	468	218	204	204
query7	4885	610	350	350
query8	353	192	168	168
query9	8757	4052	4050	4050
query10	459	359	296	296
query11	5888	2348	2142	2142
query12	161	103	101	101
query13	1287	620	437	437
query14	6264	5183	4890	4890
query14_1	4249	4238	4274	4238
query15	218	207	181	181
query16	1089	485	468	468
query17	1145	706	546	546
query18	2708	462	347	347
query19	222	184	146	146
query20	112	105	104	104
query21	230	159	137	137
query22	13525	13520	13418	13418
query23	17309	16456	16101	16101
query23_1	16276	16192	16184	16184
query24	7491	1755	1283	1283
query24_1	1280	1266	1297	1266
query25	546	435	355	355
query26	1353	370	210	210
query27	2549	613	357	357
query28	4413	1991	1957	1957
query29	1061	602	479	479
query30	344	256	244	244
query31	1129	1081	965	965
query32	111	59	61	59
query33	527	309	243	243
query34	1162	1117	640	640
query35	761	771	659	659
query36	1175	1202	1036	1036
query37	156	105	85	85
query38	1885	1699	1662	1662
query39	868	866	860	860
query39_1	827	820	831	820
query40	257	167	150	150
query41	77	74	71	71
query42	95	94	95	94
query43	327	325	287	287
query44	1455	788	775	775
query45	205	178	176	176
query46	1092	1249	734	734
query47	2111	2099	2074	2074
query48	402	418	291	291
query49	595	432	319	319
query50	1152	454	341	341
query51	10466	10521	10315	10315
query52	97	92	78	78
query53	275	287	205	205
query54	322	246	242	242
query55	76	74	70	70
query56	314	307	308	307
query57	1296	1290	1193	1193
query58	314	274	265	265
query59	1555	1588	1419	1419
query60	323	283	265	265
query61	175	168	175	168
query62	544	505	422	422
query63	238	202	212	202
query64	2971	1181	888	888
query65	4734	4558	4651	4558
query66	1769	508	389	389
query67	29240	28623	28952	28623
query68	3177	1515	965	965
query69	430	302	254	254
query70	1056	948	942	942
query71	402	355	326	326
query72	3093	2719	2278	2278
query73	826	747	443	443
query74	5096	4939	4718	4718
query75	2530	2508	2156	2156
query76	2333	1149	764	764
query77	361	391	283	283
query78	11973	11957	11355	11355
query79	1401	1112	764	764
query80	1298	538	487	487
query81	533	341	295	295
query82	639	157	118	118
query83	375	335	309	309
query84	279	153	135	135
query85	1014	621	516	516
query86	425	301	276	276
query87	1837	1821	1764	1764
query88	3725	2785	2825	2785
query89	448	384	330	330
query90	2042	204	185	185
query91	204	190	162	162
query92	66	61	61	61
query93	1707	1449	921	921
query94	721	340	311	311
query95	820	484	590	484
query96	1115	800	337	337
query97	2626	2645	2511	2511
query98	208	209	207	207
query99	1088	1110	982	982
Total cold run time: 264210 ms
Total hot run time: 176800 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 25.03 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a42d60026150be6fa921f64e362fdb388bc33e58, data reload: false

query1	0.01	0.00	0.00
query2	0.11	0.04	0.04
query3	0.26	0.13	0.13
query4	1.60	0.14	0.15
query5	0.24	0.23	0.23
query6	1.30	1.08	1.09
query7	0.03	0.01	0.00
query8	0.06	0.04	0.04
query9	0.40	0.34	0.33
query10	0.58	0.54	0.57
query11	0.20	0.13	0.14
query12	0.17	0.15	0.14
query13	0.46	0.49	0.47
query14	1.00	1.01	1.02
query15	0.62	0.60	0.60
query16	0.30	0.34	0.31
query17	1.05	1.11	1.06
query18	0.22	0.20	0.21
query19	1.99	1.91	2.01
query20	0.02	0.02	0.01
query21	15.46	0.21	0.14
query22	4.88	0.05	0.05
query23	16.14	0.30	0.13
query24	2.95	0.42	0.32
query25	0.10	0.08	0.05
query26	0.75	0.21	0.15
query27	0.04	0.03	0.05
query28	3.47	0.94	0.55
query29	12.48	4.14	3.28
query30	0.27	0.16	0.14
query31	2.78	0.59	0.31
query32	3.23	0.59	0.48
query33	3.28	3.20	3.22
query34	15.63	4.25	3.54
query35	3.54	3.51	3.57
query36	0.56	0.44	0.42
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.18	0.17	0.15
query41	0.09	0.03	0.03
query42	0.05	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 96.73 s
Total hot run time: 25.03 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants