Skip to content

[Performance] Pre-size collections with known cardinality - #18851

Merged
JackieTien97 merged 1 commit into
masterfrom
perf/presize-known-size-containers
Oct 13, 2026
Merged

JackieTien97 merged 1 commit into
masterfrom
perf/presize-known-size-containers

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Several collection materialization paths already know how many entries they will append, but allocate default-sized containers. Large batches repeatedly grow and copy ArrayList backing arrays or resize hash tables.

Pre-size 132 collections in 74 production Java files across eight modules:

  • Schema/path/plan deserialization, Pipe file-seal and handshake payloads, and progress metadata.
  • Session batch-write buffers and schema lists, and JDBC result column metadata.
  • Query projections, expression transformers, row-pattern evaluation, and spill readers.
  • Memtable tablet schema lists and recovery, TsFile device sets, ConfigNode plans/procedures, subscription retention snapshots, and UDF pattern processing.

Use serialized entry counts or the exact number of outputs produced from the input. Keep optional empty lists at capacity zero and preserve their existing null/empty handling. Hash maps and sets use the existing Guava expected-size factories, which account for the default load factor and avoid capacity arithmetic overflow.

This is independent of #18845; its WAL metadata and subscription acknowledgement changes are not included.

Validation

  • Spotless applied; Checkstyle, Spotless, and forbidden API checks passed during the reactor build.
  • English reactor test-compile passed for the eight affected modules and their dependencies (30 modules).
  • 47 existing test classes passed: 486 tests, zero failures/errors/skips. Coverage includes Session/JDBC, schema filters and paths, plan/procedure serialization, memtable partial inserts and recovery, expression/operator behavior, TsFile metadata, Pipe payloads, and UDF pattern matching.
  • Chinese locale reactor test-compile passed for the same 30 modules with -P with-zh-locale.

A separate JDK 17 allocation experiment used shared, pre-boxed integer entries and ThreadMXBean.getThreadAllocatedBytes after warm-up. For constructing and filling 1,024 entries:

Container Default capacity Pre-sized Allocation reduction
ArrayList 15,024 bytes 4,136 bytes 72.5%
HashMap 49,264 bytes 41,024 bytes 16.7%
HashSet 49,280 bytes 41,040 bytes 16.7%

These are container allocation measurements; cluster throughput has not been benchmarked.

  • Self-reviewed.

@JackieTien97
JackieTien97 merged commit a78f5a7 into master Oct 13, 2026
36 of 39 checks passed
@JackieTien97
JackieTien97 deleted the perf/presize-known-size-containers branch October 13, 2026 08:46
jt2594838 pushed a commit that referenced this pull request Oct 14, 2026
Backport #18851 by re-scanning dev/1.3 for equivalent collection
materialization paths and adapting capacity hints to this branch's
schema, plan, Session, Pipe, memtable, and UDF implementations.

(cherry picked from commit a78f5a7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants