Skip to content

[Subscription] Make column filters follow schema changes by default - #18925

Merged
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/subscription-streaming-column-filter
Oct 20, 2026
Merged

jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/subscription-streaming-column-filter

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Table subscription column filters currently bind to a schema snapshot when the topic matcher is initialized. A filter such as column_name = "s2" retains the original TAGs but omits TAGs added while the consumer is running. New matching FIELDs and tables also require rebinding.

Make column filters follow schema changes by default. Selecting part of a table's FIELDs automatically retains all its TAGs, including later additions, without ALTER TOPIC or a consumer restart. Matching FIELDs and tables are discovered dynamically, and datatype predicates use the current committed table schema. Preserve view aliases, ATTRIBUTE exclusion, per-table TIME selection, and filters that match no columns.

The expression is parsed once. Each Tablet resolves a cached table plan through the local committed schema cache; the plan is rebuilt only when that table's schema instance changes. Retain runtime TAGs in the existing pruning loop and replace per-column table lookups with direct set membership. The local schema accessor preserves metadata lease fencing and does not contact ConfigNode.

Document the behavior in iotdb-client/subscription/README.md. There is no fixed column-schema snapshot option. The topic's mode = snapshot continues to control the historical data cutoff independently of column filtering.

Validation

  • 43 unit tests passed: ColumnFilterBinderTest, ColumnFilterParserTest, TabletColumnPrunerTest, SubscriptionTopicAgentTest, DataNodeTableCacheTest, and DataNodeTableCacheLeaseTest.
  • Five changed/new methods in IoTDBSubscriptionColumnFilterIT passed: new tables, new FIELDs, new TAGs with selected FIELDs, new TAG values in TsFile payloads, and live datatype changes. Each case warms the matcher before the schema change.
  • IoTDBConsensusSubscriptionColumnFilterClusterIT#testStreamingColumnFilterIncludesNewTagsAndFieldsOnThreeDataNodes passed with 1 ConfigNode, 3 DataNodes, and data replication factor 2.
  • DataNode and integration-test compilation, Checkstyle, Spotless, and git diff --check passed.

Integration-test condition: master currently compiles CommonConfig.SUBSCRIPTION_ENABLED as false, so stock test artifacts reject topic creation. The six integration tests ran with that flag enabled only in temporary generated test artifacts, recompiling DataNodeMemoryConfig to account for the inlined constant. The original generated jars were restored afterward; the production feature gate is unchanged by this PR.

Performance

A local pruning microbenchmark compared the original snapshot matcher/pruner from the base commit with the streaming implementation and a real DataNodeTableCache. After warmup, nine alternating samples were measured per case; the table reports medians in ns/Tablet.

Columns Rows/Tablet Snapshot Streaming
8 1 1119.8 1159.1
8 1024 586.5 399.7
32 1 1760.7 1073.5
32 1024 1727.3 1057.8
128 1 7134.6 4037.6
128 1024 7335.9 4231.9

The narrow single-row case adds about 39 ns (3.5%); the other measured cases spend less time pruning. These results cover pruning with unchanged schema, not end-to-end throughput. Given the small local lookup cost and reuse of table plans, no legacy schema-snapshot configuration is retained.

This PR has:

  • been self-reviewed, including concurrent schema reads and table-plan replacement.
  • added documentation for modified behavior.
  • added comments and Javadocs for the table-plan cache and local schema accessor.
  • added or updated unit tests and integration tests.
  • been tested in IoTDB test clusters under the integration-test condition above.

@jt2594838
jt2594838 merged commit 49ba3be into apache:master Oct 20, 2026
37 of 39 checks passed
@jt2594838
jt2594838 deleted the fix/subscription-streaming-column-filter branch October 20, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants