Skip to content

Support auto-sync of tree-to-table views: detect new measurements and add the matching columns - #18874

Open
alpass163 wants to merge 2 commits into
apache:masterfrom
alpass163:cyb/auto_sync_schema
Open

alpass163 wants to merge 2 commits into
apache:masterfrom
alpass163:cyb/auto_sync_schema

Conversation

@alpass163

Copy link
Copy Markdown
Contributor

Motivation

A tree-to-table view created with auto_sync_schema = true must keep its table schema
aligned with the tree timeseries its __tree_path_pattern covers. Measurements created
after the view existed stayed invisible to the table model until the table was altered by
hand.

Design

DataNode — detection. TreeToTableViewCache indexes each view by its path prefix.
TriggerAddViewColumnVisitor runs from SchemaExecutionVisitor ahead of every schema-region
write plan: create timeseries, aligned/multi creation, the internal create nodes behind
insert into, template activation and template extension. It resolves the written devices
against the index and sends the covering views with their new measurements to ConfigNode via
the new updateTree2TableViews RPC. The trigger is best-effort: failures are logged and never
fail the write.

ConfigNode — column addition. AddViewColumnProcedure adds the columns, serialized per
table through checkDuplicateTableTask. Requests are deduplicated by queryId, so the replicas
replaying one log entry yield a single operation; overlapping ones are retried within a
bounded deadline.

Index maintenance. The index is mutated only at commit-release, always through the HA
propagator, so a rollback (CREATE OR REPLACE) restores the previous view registration
instead of leaving a stale entry.

Catch-up. Creating a view, turning auto_sync_schema on for an existing one, and extending
a schema template each scan the timeseries that already exist and add whatever columns are
missing, so a view never lags behind the tree model.

@Caideyipi Caideyipi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit f62eec46ca3f37565ca8e782f8d52d6929c0def6. The five inline findings cover a Chinese-locale compilation failure, a template-extension lock cycle, catch-up rollback for renamed source fields, stale matching metadata after view replacement, and view columns committed for rejected schema writes.

Validation:

  • Compiled the PR's cache implementation and its new tests against locally cached dependencies; all four TreeToTableViewCacheTest cases passed.
  • Ran two additional regression cases for same-path replacement with changed TAG count and changed restrict flag; both failed against this commit.
  • Compiling the Chinese ProcedureMessages.java with UTF-8 produced seven syntax errors at the conflict markers. Removing only the markers in a temporary copy also exposed the duplicated commit-release constant.

The other findings were checked through the RPC, locking, column-validation and parent/child rollback paths. A full cluster integration test was not run.

Comment on lines +2911 to +2912
TriggerAddViewColumnVisitor.addViewColumnsIfNeeded(
device2Measurements, ALTER_DEVICE_TEMPLATE_ID_PREFIX + newTemplate.getId());

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Run template-extension reconciliation after releasing the ConfigNode schema lock

For an activated template covered by an auto-sync view, this synchronous callback creates a lock cycle. ClusterSchemaManager.extendSchemaTemplate() is synchronized and waits for the DataNode UPDATE_TEMPLATE responses while holding its monitor. This DataNode handler calls updateTree2TableViews and waits for AddViewColumnProcedure, whose columnCheck() calls the synchronized tableColumnCheckForColumnExtension() on the same manager. That procedure cannot progress until template extension returns, and template extension cannot return until this callback finishes. A healthy extension therefore waits for an RPC or the 30-second procedure timeout to break the cycle, blocking other operations that need the manager lock. Schedule reconciliation after the update response, or perform it on ConfigNode outside that monitor.

Comment on lines +138 to +140
(field, type) -> {
if (Objects.isNull(table.getColumnSchema(field))) {
columns.add(toFieldColumnSchemaWithOriginName(field, type));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Compare source measurements when deciding which columns are missing

A field already mapped through FROM is treated as missing because this checks only its table column name. With only root.db.d.s1 present, creating v(device TAG, renamed INT32 FIELD FROM s1) WITH (auto_sync_schema=true) AS root.db.** makes the catch-up scan emit another column named s1. checkColumnSchemaForTree2TableView() then removes that candidate because its source is already referenced by renamed, and the empty addition fails with COLUMN_ALREADY_EXISTS. The added AddViewColumnProcedure is a child of the create procedure, so this failure triggers rollback of an otherwise valid view creation; turning auto-sync on has the same problem. Exclude fields whose source name is already represented before submitting the child, and make an already-satisfied reconciliation a success.

Comment on lines +174 to +177
final boolean added =
viewReferences
.computeIfAbsent(current, node -> new HashSet<>())
.add(new TreeToTableViewReference(database, tableName, isRestrict, numOfTags));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Replace matching metadata when a view is re-registered at the same path

TreeToTableViewReference.equals() compares only database and table names, so HashSet.add() keeps the old reference when CREATE OR REPLACE VIEW registers the same view at the same tree path. The old entry is removed above only if the path changed; changes to TAG count or restrict therefore never reach this index. For example, replacing a restricted one-TAG view at root.db.** with a restricted two-TAG view still leaves root.db.site.device uncovered, so subsequent measurements are never synchronized. Changing restrict can also leave devices incorrectly covered. Both cases fail in regression tests against this commit. Replace the existing reference for that identity even when the path is unchanged.

@alpass163
alpass163 force-pushed the cyb/auto_sync_schema branch from 51c12c8 to 11eee5f Compare October 14, 2026 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants