From f9ab05c06d3497c008db03e4514ceb70cb8c9861 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 17 Jul 2026 16:22:14 +0800 Subject: [PATCH 01/47] [doc](catalog) add hot-path heavy-op problem-class summary and fe-connector-iceberg perf audit report Generalize DORIS-27138 issue 1 (per-split whole-table planFiles() from PR #64134's getFileFormat fallback) into a problem-class doc, then audit the new fe-connector-iceberg framework against it: 7-lens multi-agent sweep + 2 adversarial verifiers per finding, 23 confirmed / 1 refuted. Top clusters: no Table-object cache (3-7 remote loadTable per planning pass), resurrected planFiles() file-format fallback (1-2 whole-table scans per query on gated tables), per-query PARTITIONS metadata-table scan for partitioned tables. Report is for review only; no code changes. Co-Authored-By: Claude Opus 4.8 (1M context) --- plan-doc/HANDOFF.md | 13 + .../perf-heavy-op-hot-path-problem-class.md | 75 + ...connector-iceberg-2026-07-17-findings.json | 3490 +++++++++++++++++ ...f-audit-fe-connector-iceberg-2026-07-17.md | 220 ++ 4 files changed, 3798 insertions(+) create mode 100644 plan-doc/perf-heavy-op-hot-path-problem-class.md create mode 100644 plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17-findings.json create mode 100644 plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index b6c60fb1c1ff6b..fac08121d87a6f 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -67,3 +67,16 @@ BE 单次启动、优雅退出（`be.out` 仅退出时 LSAN leak summary，零 ` ④ trino 改名 PR 收尾两笔（**需 release note**；BE 未跑全量构建 + fallback 无 e2e）； ⑤ 独立任务空间 `plan-doc/hive-catalog-shade-removal/`（**从它自己的 HANDOFF 进**）； ⑥ 并发 session 已结项的 QUIC 根治（`ae82ffd2573`）+ 插件包瘦身 Tier A（`dece64b9ff5`）明细。 + +--- + +# 📎 并行独立任务（与上面 CI 线无关）：热路径重操作审计（DORIS-27138 问题类） + +> 2026-07-17 独立调研，session 自包含，不影响 CI 线。用户待 review。 + +- 问题类总结（三要素 + A/B/C/D 变体 + 审计清单）：`plan-doc/perf-heavy-op-hot-path-problem-class.md` +- **fe-connector-iceberg 审计报告**（23 确认/1 驳回，分 P0/P1/P2 三层七簇）：`plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17.md` +- 完整证据 JSON（全部调用链+双路对抗验证意见）：`plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17-findings.json` +- **P0 三簇**：①无 Table 对象缓存,一次规划 3~7 次远程 loadTable；②#64134 planFiles 兜底复活（`IcebergWriterHelper.getFileFormat`,每查询 1-2 次整表扫）；③分区表每查询一次 PARTITIONS 元数据表扫描（CACHE-P1 弃二级缓存的代价）。 +- 旁获（与审计无关待单独处理）：`CreateDictionaryInfo.validateAndSet:164` 强转 `catalog.Table` ⇒ 外表 CREATE DICTIONARY 必 ClassCastException（功能缺口/潜在 bug）。 +- 下一步（等用户 review 后）：其余连接器（hive/paimon/hudi/mc）按同一问题类+同一 workflow 模式逐个审计。 diff --git a/plan-doc/perf-heavy-op-hot-path-problem-class.md b/plan-doc/perf-heavy-op-hot-path-problem-class.md new file mode 100644 index 00000000000000..55c6db711cb386 --- /dev/null +++ b/plan-doc/perf-heavy-op-hot-path-problem-class.md @@ -0,0 +1,75 @@ +# 问题类总结：热路径上的重操作放大（由 DORIS-27138 问题一泛化） + +日期：2026-07-17 +来源案例：JIRA DORIS-27138 问题一 / PR apache/doris#64134（commit `2366edffcc6`） + +## 1. 具体案例回放（用于定义问题类，非本文重点） + +PR #64134 为修复"迁移到 Iceberg 的表推断不出文件格式"，在 +`IcebergUtils.getFileFormat(table)` 里加了第三级兜底 +`inferFileFormatFromDataFiles()` → `table.newScan().planFiles()` +（= 读整表 manifest-list + 所有 manifest 的远程 IO）。 + +而这个方法的既有调用链是： + +``` +FileQueryScanNode.createScanRangeLocations() + └─ for (Split split : splits) ← per-split 循环 + └─ splitToScanRange() + └─ rangeDesc.setFormatType(getFileFormatType()) ← 每个 split 调一次 + └─ IcebergScanNode.getFileFormatType() + └─ source.getFileFormat() + └─ IcebergUtils.getFileFormat(table) + └─ [兜底] planFiles() ← 整表元数据扫描 +``` + +结果：N 个 split ⇒ N 次整表 planFiles()。对迁移表（无 format 属性），规划耗时 +从 O(planFiles) 放大到 O(N × planFiles)，split 越多越灾难。 + +## 2. 问题类定义 + +**一个重操作，被放在高乘数调用位上，且没有任何一层做 hoist / memoize。** +三个要素缺一不可： + +1. **重操作**——真实成本不小的操作： + - 远程 IO：manifest / metadata / schema / 文件列表拉取，metastore RPC； + - 大计算：整表/整分区遍历、大批量字符串处理（正则、JSON 序列化/反序列化）； + - 大对象构建：整 schema 转换、大 thrift 结构序列化。 +2. **高乘数调用位**——热路径上的循环体或高频入口： + - per-split / per-file / per-partition / per-manifest 循环（乘数动辄 10³~10⁵）； + - 每次查询规划都必经的路径（乘数 = QPS）。 +3. **无 hoist / 无 memoize**——结果对循环不变（loop-invariant）或对 + 同一次规划不变，却既没提到循环外，也没有任何缓存层兜住。 + +### 典型成因模式："伪装成轻访问器" + +方法名像 O(1) 属性读（`getFileFormat`、`getSchema`、`getXxx`），调用方从签名 +感知不到成本，于是随手放进循环。后来修 bug 的人往方法体里塞重兜底 +（#64134 就是），**所有既有调用点被静默放大**——写兜底的人不知道有多少人 +在循环里调它，循环作者也不知道 getter 变重了。本质是**接口签名不携带成本 +契约**，两个各自合理的改动叠加成事故。 + +## 3. 常见变体 + +| 变体 | 描述 | 例子 | +|---|---|---| +| A. 循环内放大 | 重操作在 per-split/per-file 循环体内被逐次调用 | 本案例：per-split × planFiles() | +| B. 单链路重复 | 一次规划链路上同一信息被多次远端获取（无循环，但串行重复 k 次） | 一次查询里反复 loadTable / 反复拉 schema | +| C. 缓存旁路 | 缓存存在，但热路径没走它，或 key 设计导致必 miss | 新建 scan 对象绕过已有 table 级缓存 | +| D. 循环不变量未提出 | 每次迭代重复计算同一个值（解析 properties、拼字符串、构建相同的 thrift 子结构） | 循环内反复 `properties.get`+解析、反复序列化同一 schema | + +A 是 B 的循环形态；C/D 是 A/B 的具体成因，单独列出便于按模式排查。 + +## 4. 审计方法（可复用清单） + +1. **枚举热路径入口**：查询规划（split 枚举、scan range 序列化）、谓词下推、 + 统计信息（row count / selectedPartitionNum）、MTMV 新鲜度、SHOW PARTITIONS、 + 写入规划与提交。 +2. **对每个循环体列出被调方法**，逐个**穿透 getter 追到底**，标注真实成本 + （内存读 / 本地计算 / 远程 IO），并判断返回值是否 loop-invariant。 +3. **对每条链路数"同一信息获取次数"**：table 加载、schema 获取、snapshot 解析、 + properties 解析各发生几次？重复的是否有缓存兜住（并确认真的命中）？ +4. **对每个缓存问 key**：热路径构造的 key 与缓存 key 是否一致？是否存在 + 每次都 miss 的 key（如含时间戳/随机量/新建对象标识）。 +5. 区分**固有成本**与**放大成本**：planFiles() 本身每次查询一次是规划的固有 + 成本，不算问题；问题是同一信息被算了 N 次或在更高乘数的位置被算。 diff --git a/plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17-findings.json b/plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17-findings.json new file mode 100644 index 00000000000000..d28047e18fca70 --- /dev/null +++ b/plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17-findings.json @@ -0,0 +1,3490 @@ +{ + "meta": { + "date": "2026-07-17", + "workflow": "iceberg-hotpath-heavy-op-audit", + "agents": 55, + "raw": 33, + "deduped": 24, + "confirmed": 23, + "refuted": 1 + }, + "confirmed": [ + { + "title": "Connector scanNodeProperties computed twice per predicated query — each compute does 2 remote loadTables + schema-dict encode; the init-time compute is thrown away wholesale", + "variant": "B-chain-redundancy", + "heavy_op": "catalog.loadTable() (metastore RPC + metadata.json read; IcebergCatalogOps line 340-341 is a raw SDK delegation with NO table-object cache — IcebergLatestSnapshotCache caches only (snapshotId, schemaId)) — invoked twice per properties compute: once via IcebergConnectorMetadata.getColumnHandles and once via IcebergScanPlanProvider.getScanNodeProperties.resolveTable; plus IcebergSchemaUtils.encodeSchemaEvolutionProp (full-schema thrift+base64) and IcebergPredicateConverter per compute", + "multiplicity": "k-times-per-query (k=2 for every query with >=1 conjunct): compute #1 at node init (FileQueryScanNode.initSchemaParams:183 getPathPartitionKeys), compute #2 after PluginDrivenScanNode.convertPredicate:795-798 blanket-nulls cachedPropertiesResult/scanNodeProperties, re-triggered by FileQueryScanNode.createScanRangeLocations:325 getFileFormatType(). Counting all sites, a predicated sync query issues ~7 remote loadTables (2 per compute x2 + streamingSplitEstimate + getSplits' buildColumnHandles at PluginDrivenScanNode:1202 + planScan's resolveTable) where legacy served one cached Table", + "entry_chain": "FileQueryScanNode.init (fe-core/.../datasource/scan/FileQueryScanNode.java:139) -> doInitialize:152 -> initSchemaParams:183 getPathPartitionKeys() -> PluginDrivenScanNode.getPathPartitionKeys (fe-core/.../datasource/scan/PluginDrivenScanNode.java:537) -> getOrLoadScanNodeProperties:1815 -> getOrLoadPropertiesResult:1776 -> buildColumnHandles:1780 -> IcebergConnectorMetadata.getColumnHandles (fe-connector-iceberg/.../IcebergConnectorMetadata.java:587 loadTable) AND getScanNodePropertiesResult:1801 -> SPI default (fe-connector-api/.../scan/ConnectorScanPlanProvider.java:455-461) -> IcebergScanPlanProvider.getScanNodeProperties (fe-connector-iceberg/.../IcebergScanPlanProvider.java:1300 resolveTable -> :1981-1993 catalogOps.loadTable). Then FileQueryScanNode.doFinalize:252 convertPredicate -> PluginDrivenScanNode.convertPredicate:795-798 invalidates -> FileQueryScanNode.createScanRangeLocations:325 getFileFormatType -> PluginDrivenScanNode:528 -> full recompute (loadTable x2 again)", + "files": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 796 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 183 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1780 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1300 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 587 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340 + } + ], + "why_heavy": "Each loadTable is remote IO: HMS getTable RPC + metadata.json fetch from object store, or a REST loadTable round-trip (no CachingCatalog wrapping anywhere in IcebergCatalogFactory, verified by grep). The compute additionally runs IcebergWriterHelper.getFileFormat (see finding 2), the full-schema field-id dict thrift/base64 encode (IcebergScanPlanProvider:1367-1421), vended-credential extraction, and predicate serialization. The memoization in getOrLoadScanNodeProperties/getOrLoadPropertiesResult is correct per-compute, but convertPredicate's blanket invalidation discards the whole init-time result — whose loadTable/format/dict/location components are loop-invariant across the two computes (only PUSHDOWN_PREDICATES_PROP depends on the conjuncts)", + "gate": "the double-compute only when the query has >=1 conjunct (convertPredicate early-returns on empty conjuncts); the 2-loadTables-per-compute is unconditional", + "est_impact": "2 wasted remote loadTables + 1 wasted schema-dict encode per predicated query (10s–100s of ms per loadTable on HMS/REST); across all planning sites ~5-7 loadTables/query vs 1 cache-hit in legacy — dominates planning latency for small queries and multiplies metastore QPS", + "confidence": "high", + "lens": "scan-planning-loop", + "dupes": [ + "convertPredicate unconditionally invalidates the scan-node-properties memo, so the whole heavy props chain (remote loadTable + schema-dict serialization + format resolution) runs twice per query with a WHERE clause [cache-audit]" + ], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Every link of the claimed chain re-derives from the actual code. (1) Heaviness: IcebergCatalogOps.loadTable (line 340-341) is a raw catalog.loadTable() SDK delegation — metastore RPC + metadata.json fetch per the cost model. grep confirms zero CachingCatalog usage in fe-connector-iceberg and fe-connector-metastore-iceberg; IcebergLatestSnapshotCache stores only (snapshotId, schemaId) (lines 54-63), never a Table; the factory's io.manifest.cache covers manifest content, not loadTable. Both entry wrappers (IcebergConnectorMetadata.loadTable:540-543 and IcebergScanPlanProvider.resolveTable:1981-1993) hit the raw seam on every call. (2) Two remote loadTables per properties compute: getOrLoadPropertiesResult (PluginDrivenScanNode:1776) calls buildColumnHandles:1780 -> metadata.getColumnHandles:1857-1859 -> IcebergConnectorMetadata.getColumnHandles:587 -> loadTable (remote #1), then getScanNodePropertiesResult:1801-1803 -> SPI default (ConnectorScanPlanProvider:455-461) -> IcebergScanPlanProvider.getScanNodeProperties:1294 -> resolveTable:1300 -> loadTable (remote #2); the compute additionally runs IcebergWriterHelper.getFileFormat:1311 (in-memory normally, but for migrated tables falls to inferFileFormatFromDataFiles -> table.newScan().planFiles() at IcebergWriterHelper:284,291 — a whole-table manifest scan, exactly the #64134 pattern) and the full-schema thrift+base64 dict encode (1367-1393). (3) k=2 multiplicity: compute #1 fires at node init (FileQueryScanNode.init:139 -> doInitialize:145/163 -> initSchemaParams:183 getPathPartitionKeys -> PluginDrivenScanNode:537-538). Then doFinalize:252 convertPredicate -> PluginDrivenScanNode:768-799: early-returns only on empty conjuncts (770-772), otherwise UNCONDITIONALLY nulls scanNodeProperties/cachedPropertiesResult (796-797) — notably iceberg has no applyFilter override at all (grep: zero hits), so applyFilter is always empty for iceberg yet the blanket invalidation still fires; the only genuinely filter-dependent output is the pushed-predicate prop (IcebergScanPlanProvider:1428+) and the filter passed onward — the 2 loadTables, format resolution, schema dict, and credential overlays are filter-invariant and are redone wholesale. Compute #2 then fires at doFinalize:253 -> createScanRangeLocations -> FileQueryScanNode:325 getFileFormatType -> PluginDrivenScanNode:527-528. The per-split calls at FileQueryScanNode:485 hit the rebuilt memo, so multiplicity is k-per-query (k=2 computes = 4 remote loadTables), as claimed — not per-split. (4) The ~7/query aggregate also checks out for a predicated sync query: 4 from the two computes + isBatchMode (FileQueryScanNode:376, memoized at PluginDrivenScanNode:1386-1389) -> streamingSplitEstimate:1414 -> resolveTable at IcebergScanPlanProvider:410 (plus manifest-list metadata reads at 424-425) + getSplits' second buildColumnHandles at PluginDrivenScanNode:1202 + planScanInternal's resolveTable at IcebergScanPlanProvider:562. No layer dedupes the Table across these sites. This is live main-path planning code for the plugin-driven iceberg scan, not dead/test-only.", + "corrected_multiplicity": "As claimed: k-times-per-query. The properties compute (2 remote loadTables + schema-dict encode each) runs exactly twice for any query with >=1 conjunct (init-time compute + post-convertPredicate recompute), once for unpredicated queries. Counting all planning sites on the synchronous path, a predicated non-sys query with batch-mode probing enabled (default) issues ~7 catalog.loadTable round-trips: 2 (compute #1) + 2 (compute #2) + 1 (streamingSplitEstimate) + 1 (getSplits buildColumnHandles) + 1 (planScan resolveTable). Not per-split: getFileFormatType at FileQueryScanNode:485 hits the memo after the compute-#2 rebuild.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "loadTable = raw catalog.loadTable(toTableIdentifier(...)) — no table-object cache; grep confirms no CachingCatalog anywhere in fe-connector-iceberg or fe-connector-metastore-iceberg" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java", + "line": 58, + "note": "cache value is only (snapshotId, schemaId) — confirms no Table caching layer exists" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1780, + "note": "getOrLoadPropertiesResult: buildColumnHandles() at 1780 (-> loadTable #1) then getScanNodePropertiesResult at 1801-1803 (-> loadTable #2); memoized only until invalidated" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 587, + "note": "getColumnHandles loads the table via loadTable(iceHandle) -> context.executeAuthenticated(() -> catalogOps.loadTable(...)) at 540-543 — remote per call" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1300, + "note": "getScanNodeProperties: resolveTable(session, iceHandle) -> ops.loadTable at 1981-1993 (raw per call); plus getFileFormat at 1311 and encodeSchemaEvolutionProp at 1376-1393 per compute" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java", + "line": 291, + "note": "inferFileFormatFromDataFiles -> table.newScan().planFiles() fallback for migrated tables (no write-format property) — a whole-table manifest scan doubled by the double-compute" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 183, + "note": "compute #1 trigger: init:139 -> doInitialize:145 -> initSchemaParams -> getPathPartitionKeys() -> PluginDrivenScanNode:537-538 -> getOrLoadScanNodeProperties" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 796, + "note": "convertPredicate blanket-nulls scanNodeProperties + cachedPropertiesResult whenever conjuncts non-empty (early return at 770-772 only for empty conjuncts); fires even though iceberg has NO applyFilter override so applyFilter is always empty" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 325, + "note": "compute #2 trigger: doFinalize:252 convertPredicate then :253 createScanRangeLocations -> getFileFormatType at 325 -> PluginDrivenScanNode:527-528 -> full recompute; per-split calls at :485 then hit the rebuilt memo" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1202, + "note": "getSplits calls buildColumnHandles AGAIN (-> loadTable #6); planScanInternal at IcebergScanPlanProvider:562 resolveTable (#7); computeBatchMode:1414 -> streamingSplitEstimate -> resolveTable at IcebergScanPlanProvider:410 (#5, memoized once via isBatchModeCache:1386)" + } + ], + "mitigation_found": "Partial mitigations exist but none covers the finding: (a) getOrLoadPropertiesResult/getOrLoadScanNodeProperties memoize within one compute generation — defeated by convertPredicate's blanket invalidation at PluginDrivenScanNode:796-797; (b) IcebergLatestSnapshotCache caches only (snapshotId, schemaId) for MVCC pinning, never the Table object; (c) io.manifest.cache (factory-derived) caches manifest file content, which softens planFiles re-reads but NOT catalog.loadTable (metastore RPC + metadata.json); (d) isBatchModeCache memoizes the batch decision so streamingSplitEstimate's loadTable happens once. No layer dedupes the ~7 loadTable calls per query.", + "fix_direction": "Two complementary hoists: (1) per-planning-pass Table memo — cache the resolved iceberg Table once per scan node / per (queryId, db.table) so IcebergConnectorMetadata.getColumnHandles, IcebergScanPlanProvider.getScanNodeProperties/streamingSplitEstimate/planScanInternal all reuse one load (a connectorSession- or handle-scoped memo, or a short-TTL Table cache in IcebergCatalogOps mirroring the legacy fe-core table cache); (2) make convertPredicate's invalidation partial — recompute only the filter-dependent output (pushed-predicate prop / filter threading) instead of nulling cachedPropertiesResult wholesale, or pass the previously computed filter-invariant props (format, path_partition_keys, schema dict, credential overlays) into the recompute. Either alone removes most of the waste; (1) also collapses the getSplits/planScan/estimate loads.", + "impact_estimate": "Per predicated query: 2 redundant remote loadTables (metastore RPC + metadata.json fetch each, typically 10s-100s ms combined on HMS/REST) + 1 redundant full-schema thrift+base64 dict encode; for migrated tables lacking write-format properties, additionally 1 redundant whole-table planFiles manifest scan. Across all planning sites, ~7 loadTables per query where a single cached load would serve — roughly 4-7x avoidable metastore/object-store QPS amplification per iceberg query, dominating planning latency for small/point queries at high QPS." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "Every hop of the candidate chain checks out against the actual code. (1) Compute #1 at init: FileQueryScanNode.init:145 -> doInitialize:163 -> initSchemaParams:183 getPathPartitionKeys() -> PluginDrivenScanNode:537-538 getOrLoadScanNodeProperties -> getOrLoadPropertiesResult:1776, which runs buildColumnHandles():1780 -> IcebergConnectorMetadata.getColumnHandles:587 -> loadTable:540-543 (remote load #1) AND getScanNodePropertiesResult:1801-1803 -> SPI default (ConnectorScanPlanProvider.java:455-461) -> IcebergScanPlanProvider.getScanNodeProperties:1300 resolveTable -> :1981-1993 ops.loadTable (remote load #2), plus IcebergWriterHelper.getFileFormat:1311 (whose no-write-format fallback at IcebergWriterHelper.java:287-300 is a whole-table planFiles — the canonical #64134 heavy op) and the full-schema thrift+base64 dict encode at :1376-1393. (2) Invalidation: FileQueryScanNode.doFinalize:252 convertPredicate -> PluginDrivenScanNode:768-799 blanket-nulls scanNodeProperties/cachedPropertiesResult at :796-797 whenever conjuncts are non-empty — even when applyFilter returned empty (iceberg does not override it; ConnectorPushdownOps.java:39-43 default Optional.empty()), because the recompute must inject the conjunct-dependent pushdown-EXPLAIN prop (:1428-1430). The loop-invariant components (2 loadTables, file format incl. possible planFiles, schema dict, location/credential overlay) are recomputed wholesale. (3) Compute #2 fires at createScanRangeLocations:325 getFileFormatType -> PluginDrivenScanNode:527-528. (4) Additional per-query loadTables the memo never covers: isBatchMode (memoized once at :1385-1390) -> streamingSplitEstimate at :1414 -> IcebergScanPlanProvider:404-425 resolveTable + remote manifest-list read (default on, ENABLE_EXTERNAL_TABLE_BATCH_MODE=true); getSplits:1202 buildColumnHandles (another getColumnHandles loadTable); planScan -> IcebergScanPlanProvider:562 resolveTable. Total ~7 remote loadTables per predicated sync query, ~5 without predicates, where 1 would suffice. Mitigation hunt (my role) found NO refuting mechanism: IcebergCatalogOps.loadTable:340-341 is a raw catalog.loadTable delegation; no CachingCatalog anywhere in fe-connector-iceberg or fe-connector-metastore-iceberg (grep); IcebergLatestSnapshotCache (default ON, 24h TTL, IcebergConnector.java:188-208) caches only (snapshotId, schemaId) for beginQuerySnapshot (IcebergConnectorMetadata:1529-1543) — it mitigates the MVCC-pin loadTable but none of the 7 chain loads; IcebergManifestCache is default OFF (IcebergConnector.java:157-162 'default off'); the SDK io.manifest.cache-enabled derivation is default-disabled (IcebergCatalogFactory.java:112-133) and would cover only manifest content, never metadata.json or the metastore RPC. The per-split memoization at :485/:527 does hold (the canonical per-split amplification is fixed), but the double-compute and the 5-7x per-query loadTable amplification have no mitigation in the default configuration. Partial mitigations exist (per-split memo, isBatchModeCache, snapshot-id cache) — all off-path for the cited loads.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 796, + "note": "convertPredicate blanket-nulls scanNodeProperties (:796) and cachedPropertiesResult (:797) for ANY query with >=1 conjunct, even when applyFilter returned empty; discards the init-time compute's loop-invariant heavy components" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1780, + "note": "each properties compute = buildColumnHandles() (loadTable via getColumnHandles) + getScanNodePropertiesResult (:1801, second loadTable via resolveTable); memo at :1777 is correct per-epoch but epoch is reset by :796-797" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 183, + "note": "compute #1 trigger: initSchemaParams -> getPathPartitionKeys (PluginDrivenScanNode:537-538); compute #2 trigger: doFinalize:252 convertPredicate then :325 getFileFormatType (PluginDrivenScanNode:527-528); per-split :485 setFormatType IS memoized (no per-split amplification)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 587, + "note": "getColumnHandles does a real remote loadTable (:540-543 raw catalogOps.loadTable inside executeAuthenticated, no memo); called per compute AND again from getSplits (PluginDrivenScanNode:1202)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1300, + "note": "getScanNodeProperties: resolveTable (:1981-1993 raw ops.loadTable), IcebergWriterHelper.getFileFormat (:1311, falls back to whole-table planFiles for tables without write-format props — IcebergWriterHelper.java:287-300), full-schema thrift+base64 dict encode (:1376-1393) — ALL rerun on the second compute; also resolveTable again in streamingSplitEstimate (:410, plus remote manifest-list read :424-425, default-enabled) and planScan (:562)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "loadTable = raw catalog.loadTable delegation (metastore RPC + metadata.json read); no Table-object cache at this seam and no CachingCatalog wrapping anywhere in fe-connector-iceberg / fe-connector-metastore-iceberg (grep verified)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 161, + "note": "mitigations found but non-refuting: latestSnapshotCache (default on, 24h TTL :188-208) caches only (snapshotId, schemaId) for beginQuerySnapshot — not Table objects; manifestCache :162 'consumed only when meta.cache.iceberg.manifest.enable is set (default off)'; SDK io.manifest.cache-enabled derivation default-disabled (IcebergCatalogFactory.java:112-133)" + } + ], + "corrected_multiplicity": "k-times-per-query: 2 full properties computes per predicated query (compute #1 at init, compute #2 after blanket invalidation), each doing 2 remote loadTables + schema-dict encode + getFileFormat (= planFiles for migrated tables lacking write-format props); counting all sites, ~7 remote loadTables per predicated sync query (2x2 computes + streamingSplitEstimate + getSplits buildColumnHandles + planScan resolveTable), ~5 without predicates, vs 1 needed. The per-split loop itself is memoized — no O(splits) amplification.", + "mitigation_found": "Partial only, none refuting: (1) getOrLoadPropertiesResult/getOrLoadScanNodeProperties memo (PluginDrivenScanNode:1776-1823) kills the per-split multiplicity but is blanket-reset by convertPredicate:796-797 on every predicated query; (2) IcebergLatestSnapshotCache (default on, 24h TTL) caches only (snapshotId, schemaId) for the MVCC pin — the 5-7 chain loadTables bypass it; (3) isBatchModeCache (:1385-1390) bounds streamingSplitEstimate to once per node; (4) IcebergManifestCache and the SDK io.manifest.cache are both default-off, and cover manifest content, not loadTable's metastore RPC / metadata.json read; (5) no CachingCatalog and no Table cache in IcebergCatalogOps/IcebergConnectorMetadata/IcebergScanPlanProvider.", + "impact_estimate": "O(k-per-query) x remote-IO x all iceberg tables: ~5 remote loadTables per query unconditionally (every loadTable = HMS getTable RPC + metadata.json object-store GET, or REST loadTable round trip; typically 10s-100s ms each), +2 more and a redundant full-schema thrift/base64 dict encode (local CPU) for every query with >=1 conjunct (the common case); for migrated tables without write-format/write.format.default properties the double compute additionally doubles a whole-table planFiles (remote manifest scan). Dominates planning latency for small/point queries and multiplies metastore QPS ~5-7x vs the single cached load the legacy IcebergMetadataCache path issued.", + "fix_direction": "Three complementary hoists/memos: (a) memoize the resolved Table per planning pass — a per-scan-node (or connector-level, snapshot-keyed) Table memo consulted by getColumnHandles/resolveTable/streamingSplitEstimate/planScan, or reinstate a table-object cache at the IcebergCatalogOps seam; (b) replace convertPredicate's blanket invalidation with a partial recompute — keep the conjunct-invariant props (file_format_type, path_partition_keys, location.*, schema dict) and rebuild only the conjunct-dependent pushdown-predicates prop; (c) pass down the column handles built at PluginDrivenScanNode:1780 for reuse at :1202 instead of a second getColumnHandles loadTable." + } + }, + { + "title": "IcebergWriterHelper.getFileFormat planFiles() fallback (the #64134 heavy op) runs on every scanNodeProperties compute — up to 2 unfiltered whole-table manifest scans per query, redundant with planScan's own enumeration", + "variant": "C-cache-bypass", + "heavy_op": "table.newScan().planFiles() with NO filter (whole-table manifest-list + manifest reads via FileIO) inside inferFileFormatFromDataFiles, just to read the first data file's format", + "multiplicity": "k-times-per-query: once per scanNodeProperties compute — i.e. 2x for predicated queries, 1x otherwise (see finding 1's double-compute at PluginDrivenScanNode.getOrLoadPropertiesResult:1776 triggered from FileQueryScanNode:183 and :325) — and it is fully redundant with the filtered planFiles that planScanInternal runs moments later on the same snapshot; no memoization keyed by (table, snapshot) at any layer", + "entry_chain": "PluginDrivenScanNode.getOrLoadScanNodeProperties (fe-core/.../PluginDrivenScanNode.java:1815) -> getScanNodePropertiesResult:1801 -> IcebergScanPlanProvider.getScanNodeProperties (fe-connector-iceberg/.../IcebergScanPlanProvider.java:1311 'file_format_type' prop) -> IcebergWriterHelper.getFileFormat (fe-connector-iceberg/.../IcebergWriterHelper.java:265) -> resolveFileFormatName:277 (both property probes miss) -> inferFileFormatFromDataFiles:287 -> table.newScan().planFiles():291", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java", + "line": 291 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1311 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 528 + } + ], + "why_heavy": "planFiles() on an unfiltered newScan() reads the manifest-list plus manifests (iceberg's ParallelIterable eagerly fans manifest readers onto the worker pool even though only the first task is consumed before close). This is the exact heavy fallback the problem-class doc canonicalizes; the per-split amplification is gone (getFileFormatType is memoized per compute — verified), but the op still sits on the every-query hot path with zero (table,snapshot)-keyed memoization, and the format it derives is trivially available from the FileScanTasks planScan enumerates anyway (each range already carries its per-file format at IcebergScanPlanProvider.buildRange:1073)", + "gate": "table.properties() contains neither 'write-format' nor 'write.format.default' — true for migrated tables AND for any table whose writer never explicitly set the property (iceberg's parquet default is implicit, not stored)", + "est_impact": "1-2 extra whole-table manifest scans per query on gated tables; for a table with thousands of manifests this is seconds of remote IO per query, every query", + "confidence": "high", + "lens": "scan-planning-loop", + "dupes": [ + "Migrated tables: getScanNodeProperties runs a second whole-scan planFiles() per query to re-infer the file format planScan already enumerates [chain-redundancy]", + "IcebergWriterHelper.getFileFormat's planFiles() fallback (ported #64134) runs 2-4 separate whole-table manifest scans per DML statement plus one redundant scan per SELECT [hidden-heavy-accessors]", + "Migrated-table file-format inference runs whole-table planFiles() on every getScanNodeProperties compute, bypassing IcebergManifestCache and redundant with planScan's own planFiles [cache-audit]", + "IcebergWriterHelper.getFileFormat's planFiles() fallback fired 3-4 times per DML statement (sink build x2 + commit x1-2), never memoized [write-commit-path]" + ], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "All three problem-class elements hold as claimed. (1) Heaviness: for a non-system iceberg table whose properties() contain neither 'write-format' nor 'write.format.default' and which has a currentSnapshot, IcebergWriterHelper.getFileFormat falls through to inferFileFormatFromDataFiles which runs an UNFILTERED table.newScan().planFiles() (IcebergWriterHelper.java:291) — remote IO reading the manifest-list, building the DeleteFileIndex (all delete manifests for v2 delete tables, paid before the first task is emitted), and eagerly fanning manifest reads onto iceberg's shared worker pool (the provider's own javadoc at IcebergScanPlanProvider.java:1999-2002 documents the ParallelIterable fan-out). Only the first task is consumed before close, so early close cancels remaining manifest reads, but manifest-list + delete-index + in-flight prefetch are paid per call; iceberg manifest caching is default-disabled (IcebergCatalogFactory.java:68 DEFAULT_MANIFEST_CACHE_ENABLE=false). (2) Multiplicity: I independently traced two computes per predicated query per iceberg scan node. Compute #1 during Nereids init: FileQueryScanNode.init:139→doInitialize:145→initSchemaParams:163→getPathPartitionKeys call at :183→PluginDrivenScanNode.getPathPartitionKeys:537-538→getOrLoadPropertiesResult:1776 (cache empty)→provider :1801-1803→SPI default getScanNodePropertiesResult (ConnectorScanPlanProvider.java:455-462, iceberg does not override)→IcebergScanPlanProvider.getScanNodeProperties:1294→file_format_type at :1310-1311→getFileFormat→planFiles. Then FileQueryScanNode.doFinalize:252 calls PluginDrivenScanNode.convertPredicate:768 which, whenever conjuncts are non-empty (early return only at :770-772), unconditionally nulls scanNodeProperties/cachedPropertiesResult at :796-798 (required because other props are filter-dependent); compute #2 immediately follows at createScanRangeLocations→getFileFormatType (FileQueryScanNode.java:325)→PluginDrivenScanNode:527-528→full recompute→second unfiltered planFiles on the same snapshot. Per-split calls at FileQueryScanNode.java:485 hit the warm memo (:1815-1822) — the candidate correctly says per-split amplification is gone. (3) No hoist/memoize: getFileFormat is a static uncached helper (no callers cache it; the scan-path caller is :1311); the format is filter-invariant yet lives inside the filter-invalidated blob, and the same information is redundantly available from planScanInternal's own FILTERED planFiles (IcebergScanPlanProvider.java:1657/1660) whose per-file format travels per range (dataFile.format() at :1073). This is the exact heavy fallback the problem-class doc canonicalizes (#64134), now at k=1-2 per query instead of per-split — variant B/C (single-chain repeat + cache bypass via invalidation), on the every-query planning hot path for gated tables.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java", + "line": 291, + "note": "inferFileFormatFromDataFiles: try(...planFiles()) unfiltered whole-table scan, consumes only first FileScanTask; reached via getFileFormat:265->resolveFileFormatName:277 when both 'write-format':278 and 'write.format.default':281 probes miss; currentSnapshot==null short-circuits to parquet at :288" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1311, + "note": "getScanNodeProperties puts file_format_type = IcebergWriterHelper.getFileFormat(table) for every non-system table; runs on EVERY scanNodeProperties compute; no caching of the result anywhere" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 796, + "note": "convertPredicate unconditionally nulls scanNodeProperties+cachedPropertiesResult (796-798) whenever conjuncts non-empty (early return only at 770-772) — the deliberate cache bypass that forces compute #2" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1776, + "note": "getOrLoadPropertiesResult: single null-checked memo field; provider call at 1801-1803; getOrLoadScanNodeProperties memo at 1815-1822 protects the per-split getFileFormatType calls (527-528) — per-split amplification confirmed absent" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 183, + "note": "compute #1 trigger: init:139->doInitialize:145->initSchemaParams:163 calls getPathPartitionKeys at :183 -> PluginDrivenScanNode.getPathPartitionKeys:537-538 -> first getScanNodePropertiesResult before any conjuncts exist" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 325, + "note": "compute #2 trigger: doFinalize:252 convertPredicate (invalidates) then :253 createScanRangeLocations -> getFileFormatType at :325 -> recompute with filtered handle; per-split setFormatType(getFileFormatType()) at :485 hits warm memo" + }, + { + "file": "fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java", + "line": 455, + "note": "SPI default getScanNodePropertiesResult wraps getScanNodeProperties; only ES overrides it, so the iceberg chain routes through getScanNodeProperties:1294 exactly as claimed" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1073, + "note": "redundancy: buildRange reads dataFile.format() per file from planScanInternal's own FILTERED planFiles (:1657/:1660) moments later on the same snapshot — the format info the fallback derives is enumerated anyway" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1999, + "note": "provider's own javadoc: scan planning reads manifest-list+manifests via table.io(), fanned onto iceberg's shared worker pool (ParallelIterable) for multi-manifest tables — supports the eager-fan-out heaviness claim" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE = false — iceberg io.manifest.cache is opt-in, so by default every planFiles fallback pays real remote IO" + } + ], + "corrected_multiplicity": "k-times-per-query per iceberg scan node, k in {1,2}: exactly 1 compute (at Nereids init via getPathPartitionKeys) for queries whose scan node has no conjuncts, exactly 2 (init + post-convertPredicate-invalidation recompute at createScanRangeLocations) when conjuncts exist. Multiple iceberg scan nodes in one query multiply accordingly. Per-split calls are memoized — the candidate's claim is exactly right, no correction needed.", + "mitigation_found": "Partial mitigations exist but none covers the finding: (a) fe-core's cachedPropertiesResult/scanNodeProperties memo protects per-split calls, but is deliberately invalidated in convertPredicate:796-798 because OTHER properties are filter-dependent — the filter-INVARIANT format resolution is collateral damage; (b) inferFileFormatFromDataFiles consumes only the first FileScanTask and closes early, canceling pending ParallelIterable manifest reads — bounds but does not eliminate the IO (manifest-list read + DeleteFileIndex over all delete manifests + in-flight prefetch still paid); (c) iceberg io.manifest.cache exists but is default-disabled (opt-in via meta.cache.iceberg.manifest.enable). No (table,snapshot)-keyed memo of the format at any layer.", + "fix_direction": "Hoist the filter-invariant format out of the filter-invalidated blob. Cheapest connector-local fix: memoize the resolved format name in IcebergScanPlanProvider keyed by (table identity, currentSnapshot.snapshotId()) — a tiny Caffeine/ConcurrentHashMap entry — so compute #2 and subsequent queries on the same snapshot skip the planFiles fallback entirely. Alternatively (or additionally) resolve the format lazily from the FIRST FileScanTask of planScanInternal's own filtered planFiles instead of a separate unfiltered scan; note the scan-level file_format_type must still be known before split fetch (BE selects FileScannerV2 vs V1 from it — see the comment at IcebergScanPlanProvider.java:1303-1309), so the value must be produced at properties time, which favors the snapshot-keyed memo. A narrower fe-core option — splitting ScanNodePropertiesResult into filter-dependent and filter-invariant parts so convertPredicate only invalidates the former — also works but touches the generic SPI surface.", + "impact_estimate": "On gated tables (no write-format/write.format.default property — common for migrated tables and any table whose creator relied on the implicit parquet default — and non-empty): 1-2 extra unfiltered planFiles per iceberg scan node per query, every query, on the planning hot path. Per call the guaranteed cost is manifest-list fetch + full DeleteFileIndex construction (reads ALL delete manifests for v2 tables with deletes before the first task) + a pool-width burst of manifest reads before early close; on object storage that is roughly tens-to-hundreds of ms for small tables and hundreds of ms to low seconds for large or delete-heavy snapshots — doubled for predicated queries. The candidate's 'seconds per query for thousands of manifests' is plausible for delete-heavy v2 tables, slightly overstated for pure-append tables where early close cancels most manifest reads." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The heavy op and its multiplicity are real and unmitigated in the default configuration. Chain verified: FileQueryScanNode.init() -> initSchemaParams (getPathPartitionKeys, line 183) triggers compute #1 of scanNodeProperties; PluginDrivenScanNode.convertPredicate() (called from doFinalize at FileQueryScanNode:252) unconditionally nulls scanNodeProperties/cachedPropertiesResult at lines 796-797 whenever the query has any conjunct — even when applyFilter returns empty; createScanRangeLocations (FileQueryScanNode:325 getFileFormatType) then triggers compute #2. Each compute runs IcebergScanPlanProvider.getScanNodeProperties (via the SPI default getScanNodePropertiesResult wrapper, not overridden), which does its OWN remote catalog.loadTable (resolveTable -> IcebergCatalogOps.loadTable -> catalog.loadTable directly; no CachingCatalog, no table-object cache — grep across fe-connector-iceberg and fe-connector-metastore-iceberg found none), then IcebergWriterHelper.getFileFormat -> inferFileFormatFromDataFiles -> unfiltered table.newScan().planFiles() when neither 'write-format' nor 'write.format.default' is in table.properties() (iceberg does not persist the parquet default, so this gate is true for any table whose writer never explicitly set it — far broader than just migrated tables). Mitigation hunt results, all negative for the default config: (1) the cachedPropertiesResult memo (PluginDrivenScanNode:1776-1809) does kill the canonical per-split amplification — getFileFormatType per split hits the memo — but it is deliberately invalidated by convertPredicate, leaving k=2 computes for predicated queries; (2) the iceberg SDK manifest content cache (io.manifest.cache-enabled) is derived default-FALSE (IcebergCatalogFactory DEFAULT_MANIFEST_CACHE_ENABLE=false, legacy parity); (3) the connector's IcebergManifestCache is gated by the same default-off meta.cache.iceberg.manifest.enable flag AND is only consulted on the manifest-level planning path — the infer path's raw table.newScan().planFiles() bypasses it regardless; (4) because each compute loads a FRESH Table (fresh BaseSnapshot objects), not even the manifest-list read is amortized across the two computes or with planScanInternal's later filtered planFiles; (5) IcebergLatestSnapshotCache caches only snapshot-id pins for MVCC, not manifests or format; (6) no (table,snapshot)-keyed memo of getFileFormat exists anywhere in the connector. The redundancy claim also holds: planScan's enumeration already knows each file's format (buildRange reads dataFile.format() at IcebergScanPlanProvider:1073). One correction to the candidate's cost model: the infer scan is early-terminated (first FileScanTask, then close), so on append-only tables it is not a literal whole-table manifest read — per compute it costs one remote loadTable + one manifest-list read + an eager worker-pool-width burst of manifest reads (ParallelIterable fans out before close cancels); on v2 merge-on-read tables with delete files, DeleteFileIndex construction reads ALL delete manifests before the first task is emitted, which IS whole-table-scale remote IO. Each compute also pays the remote loadTable even when the format properties ARE set (that part is un-gated), reinforcing that the convertPredicate cache reset recomputes filter-invariant values.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java", + "line": 291, + "note": "inferFileFormatFromDataFiles: try (CloseableIterable files = table.newScan().planFiles()) — unfiltered scan, consumes only the first task; reached via resolveFileFormatName:277-284 only when both 'write-format' and 'write.format.default' are absent from table.properties()" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1311, + "note": "getScanNodeProperties puts file_format_type = IcebergWriterHelper.getFileFormat(table) for every non-system-table compute; table comes from resolveTable(session, iceHandle) at line 1300" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1985, + "note": "resolveTable -> catalogOpsResolver.apply(session) -> ops.loadTable(db, table): a fresh remote load per call, no memo field on the provider; IcebergCatalogOps default impl delegates straight to catalog.loadTable (IcebergCatalogOps.java:340-341); no CachingCatalog wrap found in fe-connector-iceberg or fe-connector-metastore-iceberg" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 796, + "note": "convertPredicate ends with 'scanNodeProperties = null; cachedPropertiesResult = null;' — executed whenever conjuncts is non-empty (the reset sits OUTSIDE the result.isPresent() block), forcing a second full getScanNodeProperties compute for every predicated query" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1776, + "note": "getOrLoadPropertiesResult memoizes into cachedPropertiesResult — this is the layer that kills the canonical per-split amplification (getFileFormatType:527-534 and getPathPartitionKeys:537-544 both read the memo), but it does not survive the convertPredicate reset" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 183, + "note": "initSchemaParams (called from init():163) calls getPathPartitionKeys() — trigger of compute #1 during node init, before predicate pushdown exists" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 252, + "note": "doFinalize(): convertPredicate() at :252 (cache reset) then createScanRangeLocations() at :253 whose getFileFormatType() call at :325 triggers compute #2" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE = false — the io.manifest.cache-enabled derivation (appendManifestCacheProperties:119-131) only turns the SDK manifest content cache on when the user sets meta.cache.iceberg.manifest.enable; default config gets no manifest-read caching" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java", + "line": 47, + "note": "javadoc: consumed only by the manifest-level planning path 'gated by meta.cache.iceberg.manifest.enable, default off — the default scan path is the iceberg SDK planFiles()'; the infer path's raw table.newScan().planFiles() bypasses this cache in all configurations" + }, + { + "file": "fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java", + "line": 455, + "note": "default getScanNodePropertiesResult wraps getScanNodeProperties; IcebergScanPlanProvider does not override it, so PluginDrivenScanNode:1801-1803 lands directly on the chain above" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1073, + "note": "buildRange already derives each range's format from dataFile.format() during planScan's enumeration — confirming the scan-level infer planFiles is redundant with information the planning pass produces anyway" + } + ], + "mitigation_found": "Partial only, none effective by default: (1) PluginDrivenScanNode.getOrLoadPropertiesResult memoization (line 1776) removes the canonical per-split amplification, but convertPredicate (lines 796-797) invalidates it for every predicated query, so the heavy compute runs twice; (2) iceberg SDK manifest content cache exists but DEFAULT_MANIFEST_CACHE_ENABLE=false (IcebergCatalogFactory.java:68, legacy parity) — off by default; (3) connector IcebergManifestCache is gated by the same default-off flag and is bypassed by the infer path's raw table.newScan().planFiles() even when enabled; (4) no CachingCatalog/table-object cache — each compute does its own remote catalog.loadTable and gets fresh Snapshot objects, so the manifest-list read is not amortized either; (5) IcebergLatestSnapshotCache caches only MVCC snapshot-id pins, irrelevant to format/manifests. No (table,snapshot)-keyed memo of getFileFormat exists at any layer.", + "corrected_multiplicity": "k-times-per-query with k=1 (no conjuncts on the scan) or k=2 (any conjunct triggers the convertPredicate cache reset) — NOT per-split (memo verified). Each of the k computes = 1 remote catalog.loadTable + 1 manifest-list read + a worker-pool-width burst of manifest reads (early-closed after the first FileScanTask), plus ALL delete-manifest reads (DeleteFileIndex) on v2 merge-on-read tables. The candidate's 'whole-table manifest scan' wording is exact only for the delete-manifest side; the data-manifest side is a truncated-but-eager scan.", + "impact_estimate": "O(k-per-query, k<=2) x remote-IO (loadTable + manifest-list + partial/eager manifest fan-out; full delete-manifest read on MoR tables) x broad trigger: every iceberg table whose properties lack BOTH 'write-format' and 'write.format.default' — iceberg never persists the implicit parquet default, so this covers most Spark/Flink-created tables, not just migrated ones — on every query, in the default configuration. Hundreds of ms to seconds of extra planning latency per query on large tables (worst on MoR tables with many delete manifests); doubles for predicated queries.", + "fix_direction": "Two complementary hoists: (a) stop recomputing filter-invariant properties — convertPredicate's blanket reset (PluginDrivenScanNode:796) should not discard file_format_type/path_partition_keys, which do not depend on conjuncts (split the ScanNodePropertiesResult into filter-invariant and filter-dependent parts, or recompute only the latter); (b) memoize/pass-down the format at the connector: key IcebergWriterHelper.getFileFormat by (table identity, snapshotId) in a small per-catalog cache, or better, derive the scan-level format from the first planned FileScanTask that planScanInternal enumerates anyway (per-file format already flows at buildRange:1073), eliminating the separate unfiltered planFiles entirely. Reusing one resolveTable result per planning pass (handle- or session-scoped) would also remove the duplicate remote loadTable." + } + }, + { + "title": "REST vended-credentials scans rebuild the full StorageProperties map (incl. hadoop Configuration) per data file AND per delete file in normalizeUri", + "variant": "D-loop-invariant-not-hoisted", + "heavy_op": "DefaultConnectorContext.buildVendedStorageMap -> CredentialUtils.filterCloudStorageProperties + StorageProperties.createAll (iterates all providers, initNormalizeAndCheckProps + buildHadoopStorageConfig = hadoop Configuration construction + per-key set) executed inside every normalizeStorageUri(rawUri, vendedToken) call", + "multiplicity": "per-split and per-delete-file: planScanInternal loop (IcebergScanPlanProvider.java:627-637) -> buildRangeForTask:689 -> buildRange:1105 normalizeUri (1 per data-file range) + buildDeleteFiles:1129 -> convertDelete:1157 normalizeUri (1 per merge-on-read delete file per task); identical on the streaming path (IcebergStreamingSplitSource.hasNext:517 -> buildRangeForTask) and $position_deletes (buildPositionDeleteRange:874). 10^3–10^5+ calls per scan", + "entry_chain": "PluginDrivenScanNode.getSplits (fe-core/.../PluginDrivenScanNode.java:1242 planScan) -> IcebergScanPlanProvider.planScanInternal (fe-connector-iceberg/.../IcebergScanPlanProvider.java:627-637 loop) -> buildRangeForTask:699 -> buildRange:1105 normalizeUri -> :1222-1224 context.normalizeStorageUri -> TcclPinningConnectorContext:166-167 -> DefaultConnectorContext.normalizeStorageUri (fe-core/.../connector/DefaultConnectorContext.java:392-409) -> buildVendedStorageMap:225-242 -> StorageProperties.createAll (fe-core/.../datasource/property/storage/StorageProperties.java:137-160) -> buildHadoopStorageConfig:317-324", + "files": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java", + "line": 405 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1105 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1157 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/StorageProperties.java", + "line": 137 + } + ], + "why_heavy": "StorageProperties.createAll runs every provider's guess/parse, initNormalizeAndCheckProps (endpoint/region derivation, validation) and buildHadoopStorageConfig (a hadoop Configuration object build + user-fs key copies) — milliseconds-scale CPU + allocation per call. The vendedToken input is loop-invariant (extracted ONCE per scan at IcebergScanPlanProvider.planScanInternal:586-587 / streamSplits:458-459), so the derived typed map is identical for every file in the scan, yet it is rebuilt inside every call; the static-map path is properly cached (CatalogProperty.initStorageProperties:167-187 double-checked), only the vended arm bypasses all caching", + "gate": "REST catalog with iceberg.rest.vended-credentials-enabled=true (vendedToken non-empty; empty token short-circuits at buildVendedStorageMap:227-229). Merge-on-read tables multiply the count by deletes-per-task", + "est_impact": "O(N_files + N_deletes) hadoop-Configuration builds per scan: at 50k files ~ tens of seconds of pure FE CPU added to split generation on vended REST catalogs", + "confidence": "high", + "lens": "scan-planning-loop", + "dupes": [ + "normalizeStorageUri rebuilds the vended StorageProperties map (createAll + Hadoop config) per split and per delete file, though the raw token is loop-invariant and already hoisted [per-split-serialization]" + ], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Independently re-derived the full chain. Heaviness holds: DefaultConnectorContext.normalizeStorageUri(rawUri, token) calls buildVendedStorageMap on EVERY invocation (line 405) with no caching at any layer; with a non-empty vended token this runs CredentialUtils.filterCloudStorageProperties + StorageProperties.createAll, which iterates the full provider registry with guessIsMe heuristics, appends a default HdfsProperties, and for every matched provider runs initNormalizeAndCheckProps (endpoint/region derivation + validation) and buildHadoopStorageConfig -> initializeHadoopStorageConfig -> `new Configuration()` (loadDefaults=true, per-instance default-resource XML parse) + per-key sets. For an AWS vended token that is at least 2 hadoop Configuration builds per call (S3Properties + default HdfsProperties) — millisecond-class CPU + allocation, a moderate-CPU heavy op per the class doc. Multiplicity holds as claimed or worse: the vended token is extracted exactly ONCE per scan (loop-invariant, IcebergScanPlanProvider 586-587 eager / 458-459 streaming / 843-844 position-deletes), yet the typed map is rebuilt inside every normalizeUri call: once per FileScanTask (per-SPLIT, after TableScanUtil.splitFiles slicing) via buildRange:1105, PLUS once per merge-on-read delete file per task via buildDeleteFiles:1129-1137 -> convertDelete:1157 (a delete file referenced by k tasks is re-normalized k times), identically on the streaming path (hasNext:516-518 -> buildRangeForTask) and the $position_deletes path (:874). Entry is the live production hot path: PluginDrivenScanNode.getSplits -> scanProvider.planScan (PluginDrivenScanNode.java:1243) -> planScanInternal loop 627-637. No mitigation exists on the vended arm; the contrast claim is also correct — the static-map arm IS cached (CatalogProperty.initStorageProperties double-checked at 167-179), only the vended arm bypasses caching. TcclPinningConnectorContext:166-167 is a pure delegate (no memoization); note it lives in fe-connector-iceberg, not fe-core as the candidate's entry_chain said — minor path erratum, does not affect the finding. Gate confirmed and real: REST flavor + iceberg.rest.vended-credentials-enabled=true (restVendedCredentialsEnabled:1243-1246); empty token short-circuits at buildVendedStorageMap:227-229 so non-vended catalogs pay nothing.", + "corrected_multiplicity": "per-split (one normalizeUri per FileScanTask after splitFiles slicing — more than per-file for large files) + per-delete-file-reference per task (merge-on-read: each task re-normalizes every attached delete file, shared delete files re-processed k times) + per position-delete range on $position_deletes; identical on eager (planScanInternal:627-637) and streaming (IcebergStreamingSplitSource.hasNext:516-518) paths. Each call = full StorageProperties.createAll incl. >=2 hadoop Configuration builds.", + "mitigation_found": "None on the vended arm: buildVendedStorageMap (DefaultConnectorContext.java:225-242) has no cache, TcclPinningConnectorContext:166-167 is a pure passthrough, and IcebergScanPlanProvider.normalizeUri:1222-1223 calls through per path. Only the static-map arm is cached (CatalogProperty.initStorageProperties double-checked locking, CatalogProperty.java:167-179), and the empty-token short-circuit (buildVendedStorageMap:227-229) protects non-vended catalogs only.", + "fix_direction": "Hoist the loop-invariant map build to once per scan. The token is already extracted once per scan in the provider, so the seam is DefaultConnectorContext: either (a) memoize buildVendedStorageMap with a single-entry equals-keyed cache (token map -> typed StorageProperties map) — tokens are stable within a scan and per-table, so hit rate is ~100% within a scan; or (b) add a ConnectorContext API returning a per-scan URI-normalizer object that resolves the vended map once (the connector cannot build fe-core StorageProperties itself, so the resolved map must stay engine-side). Option (a) is the minimal surgical fix requiring no SPI change; same fix should cover getBackendFileType (DefaultConnectorContext.java:412-422) which shares the per-call rebuild.", + "impact_estimate": "O(N_splits + N_delete_file_refs) full StorageProperties.createAll runs per scan on vended REST catalogs: each run does provider guessing + validation + >=2 `new Configuration()` builds (per-instance hadoop default-resource XML parse, ~1-5 ms typical). At 10^4 splits that is tens of seconds of pure FE CPU added to split enumeration; at 10^5 splits (or merge-on-read heavy tables where delete refs multiply calls) minutes-scale, plus significant allocation/GC pressure on the planning thread. Candidate's estimate confirmed at order of magnitude.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java", + "line": 405, + "note": "normalizeStorageUri(rawUri, rawVendedCredentials) calls buildVendedStorageMap on every invocation before LocationPath.of; no cache" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java", + "line": 235, + "note": "buildVendedStorageMap runs filterCloudStorageProperties + StorageProperties.createAll per call; empty-token short-circuit at 227-229 is the only guard" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/StorageProperties.java", + "line": 155, + "note": "createAll: for each matched provider (plus default HdfsProperties inserted at 151-153) runs initNormalizeAndCheckProps + buildHadoopStorageConfig; buildHadoopStorageConfig at 317-324 builds hadoop Configuration + per-key sets" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/AbstractS3CompatibleProperties.java", + "line": 265, + "note": "initializeHadoopStorageConfig does `new Configuration()` (loadDefaults) per StorageProperties instance — the concrete heavy allocation/parse" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 587, + "note": "vendedToken extracted ONCE per scan (loop-invariant); streaming twin at 458-459, position-deletes twin at 843-844" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 631, + "note": "eager per-task loop 627-637 -> buildRangeForTask -> buildRange:699-700" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1105, + "note": "buildRange: normalizeUri(rawDataPath, vendedToken) — one per data-file split range" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1157, + "note": "convertDelete: normalizeUri per merge-on-read delete file per task (via buildDeleteFiles loop 1129-1137 from buildRange:1117)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 517, + "note": "streaming path IcebergStreamingSplitSource.hasNext calls buildRangeForTask per task with the same captured vendedToken — identical per-split cost" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1223, + "note": "provider normalizeUri delegates straight to context.normalizeStorageUri(rawPath, vendedToken); no provider-side memoization" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java", + "line": 166, + "note": "2-arg normalizeStorageUri is a pure delegate (166-167), no caching; note: file is in fe-connector-iceberg, not fe-core as candidate stated (minor erratum)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1243, + "note": "gate: restVendedCredentialsEnabled = REST flavor AND iceberg.rest.vended-credentials-enabled=true; extractVendedToken:1259-1271 returns FileIO props + vended credentials (non-empty for vended REST)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java", + "line": 167, + "note": "contrast confirmed: static storage map IS cached via double-checked initStorageProperties (167-179); only the vended arm rebuilds per call" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1243, + "note": "production entry: onPluginClassLoader(... scanProvider.planScan(...)) — the audited loop is on the live query-planning hot path" + } + ] + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "Every claim in the candidate checks out and no mitigation exists at any layer for the vended arm. The vended token is extracted exactly once per scan (IcebergScanPlanProvider.java:586-587 sync, :458-459 streaming) and is loop-invariant (the SAME Map instance is threaded through every call), yet each normalizeUri call — one per data-file range (:1105), one per merge-on-read delete file per task (:1157 via :1117/:1135-1136), one per position-delete split (:874), identically on the streaming path (:516-518) — reaches DefaultConnectorContext.normalizeStorageUri(:392-409) which unconditionally rebuilds the full typed storage map via buildVendedStorageMap(:405 -> :225-242): CredentialUtils prefix filter + StorageProperties.createAll(:137-160) = 12-provider guessIsMe pass, a default HdfsProperties always inserted (:151-153) PLUS the matched S3-family provider, each running reflection property binding, initNormalizeAndCheckProps, and buildHadoopStorageConfig(:317-332) which constructs a fresh hadoop Configuration per provider (HdfsProperties.java:134, AbstractS3CompatibleProperties.java:265) and iterates the whole user prop map. Mitigation hunt was exhaustive and negative: TcclPinningConnectorContext:166-167 is a pure delegate; DefaultConnectorContext caches only catalogFileSystem (:103), not the vended map; fe-connector-cache has zero StorageProperties references; LocationPath.ofWithCache(:232) caches only scheme/fsId strings and is not used here; CredentialUtils is stateless. The STATIC arm proves the intent: CatalogProperty.storagePropertiesMap is double-checked-cached (CatalogProperty.java:167-179), and an empty token short-circuits to it (:227-229) — only the vended arm bypasses all caching. Gate is real but standard for the feature: REST flavor + iceberg.rest.vended-credentials-enabled=true (restVendedCredentialsEnabled:1243-1246); non-vended catalogs are unaffected. This is variant D (loop-invariant not hoisted): the identical typed map is derivable once per scan but is rebuilt O(ranges + delete-refs) times on the planning hot path (entry: PluginDrivenScanNode.java:1242-1244 planScan, once per query).", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 586, + "note": "vendedToken extracted ONCE per scan (loop-invariant; same Map instance passed to every buildRangeForTask)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 631, + "note": "per-FileScanTask loop (628-637) calls buildRangeForTask with vendedToken; streaming twin at hasNext():516-518 with token captured once at :458-459" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1105, + "note": "buildRange: normalizeUri(rawDataPath, vendedToken) once per data-file range; :1117 buildDeleteFiles -> :1135-1136 loop -> convertDelete:1157 normalizeUri once per delete file per task; $position_deletes at :874" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1222, + "note": "normalizeUri helper is a straight per-call delegate to context.normalizeStorageUri(rawPath, vendedToken) — no connector-side memoization of the derived map" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java", + "line": 166, + "note": "wrapper is a pure delegate (166-167), adds no caching" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java", + "line": 405, + "note": "2-arg normalizeStorageUri calls buildVendedStorageMap on EVERY invocation; buildVendedStorageMap (225-242) has no cache field — the class caches only catalogFileSystem (:103); empty token short-circuits at 227-229 to the static arm" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/StorageProperties.java", + "line": 137, + "note": "createAll per call: 12-provider guessIsMe pass (:142-147), default HdfsProperties always inserted (:151-153), then per provider initNormalizeAndCheckProps + buildHadoopStorageConfig (:155-158); buildHadoopStorageConfig (:317-332) also iterates the full user prop map" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/HdfsProperties.java", + "line": 134, + "note": "new hadoop Configuration() per HdfsProperties build; same per S3-family at AbstractS3CompatibleProperties.java:265 — so >=2 hadoop Configuration constructions + reflection binding (S3Properties.java:195) per normalized path" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogProperty.java", + "line": 167, + "note": "CONTRAST/existing mitigation for static arm only: storagePropertiesMap is volatile double-checked cached (167-179); the vended arm bypasses this entirely" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1242, + "note": "entry: planScan invoked once per query under onPluginClassLoader — the amplification is entirely inside the connector's per-task/per-delete loops" + } + ], + "mitigation_found": "None for the vended arm, at any layer. Checked: TcclPinningConnectorContext (pure delegate), DefaultConnectorContext (caches only catalogFileSystem; buildVendedStorageMap stateless), connector-side normalizeUri helper (no memo), CredentialUtils (stateless), LocationPath (ofWithCache exists but caches only scheme/fsId strings and is unused here), fe-connector-cache (zero StorageProperties references), CatalogProperty (double-checked cache exists but ONLY for the static storage map, which the non-empty-token path replaces). Partial mitigations that bound but do not fix it: empty-token short-circuit (buildVendedStorageMap:227-229) makes non-vended catalogs use the cached static map, and fail-soft catch (:238-241) prevents errors but not cost.", + "corrected_multiplicity": "O(N_data_ranges + SUM over tasks of |task.deletes()| + N_position_delete_splits) per query — per-split plus per-(task x delete-file); a delete file referenced by k tasks is normalized k times. Candidate's multiplicity is accurate; minor line corrections: the sync-path call into buildRangeForTask is at IcebergScanPlanProvider.java:631 (:689 is the method signature) and buildDeleteFiles is invoked at :1117 (:1129 is the signature).", + "impact_estimate": "O(splits + delete-file refs) x local-CPU/allocation (no remote IO: >=2 hadoop Configuration builds + 12-provider probe + reflection binding + full prop-map iterations per call, roughly 0.1-2 ms each) x trigger breadth: ONLY REST catalogs with iceberg.rest.vended-credentials-enabled=true and a non-empty extracted token — but for those catalogs it fires on every query and every file, both eager and streaming paths, worst on merge-on-read (deletes multiply the count). At 10k-100k ranges this adds seconds to tens of seconds of serialized FE planning CPU per query, plus significant allocation/GC pressure; non-vended catalogs are unaffected (cached static map).", + "fix_direction": "Hoist/memoize the token-to-typed-map derivation to once per scan. Least invasive given the SPI split: memoize inside DefaultConnectorContext (token map equality- or identity-keyed single-entry cache, since the connector passes the SAME Map instance for a whole scan) so both normalizeStorageUri(String,Map) and getBackendFileType hit it — mirrors the existing CatalogProperty static-arm cache and keeps property parsing in fe-core/engine side per the no-parsing-in-connector rule. Alternative: add a ConnectorContext seam that resolves the vended token once into an opaque normalized-storage handle that the connector threads through buildRangeForTask instead of the raw token." + } + }, + { + "title": "streamingSplitEstimate performs a redundant extra loadTable + buildScan + manifest-list read on every iceberg query before planScan repeats the same work", + "variant": "B-chain-redundancy", + "heavy_op": "resolveTable -> catalogOps.loadTable (remote metastore RPC + metadata.json) plus snapshot.dataManifests(table.io()) (manifest-list file read from object store), executed inside the isBatchMode gate computation", + "multiplicity": "k-times-per-query (k=1, memoized via isBatchModeCache — verified at PluginDrivenScanNode.isBatchMode:1385-1390): every query pays one loadTable + one manifest-list read that planScan (sync path) or streamSplits (batch path) immediately redoes on its own freshly re-loaded Table; nothing (Table object, snapshot, manifest list) is shared between the two", + "entry_chain": "FileQueryScanNode.createScanRangeLocations (fe-core/.../FileQueryScanNode.java:376 isBatchMode) -> PluginDrivenScanNode.isBatchMode:1385 -> computeBatchMode:1414 -> IcebergScanPlanProvider.streamingSplitEstimate (fe-connector-iceberg/.../IcebergScanPlanProvider.java:404) -> resolveTable:410 (loadTable) + buildScan:411 + snapshot.dataManifests(table.io()):424-425; then the sync path getSplits -> planScan -> planScanInternal:562 resolveTable (another loadTable) -> planFileScanTask:627 planFiles (re-reads the same manifest list)", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 410 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 424 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1414 + } + ], + "why_heavy": "loadTable is a full remote table load (see finding 1 — no table-object cache); dataManifests(io) fetches and parses the snapshot's manifest-list file. The estimate itself (summing addedFilesCount/existingFilesCount from manifest-list entries) is cheap by design, but its two remote prerequisites duplicate work the immediately-following planScan/streamSplits performs against its OWN resolveTable result — a Table/scan handoff or a per-query Table memo would remove both", + "gate": "enable_external_table_batch_mode=true (session default true) and non-system-table; runs on every normal iceberg query regardless of whether streaming is chosen", + "est_impact": "+1 remote loadTable +1 manifest-list read on every iceberg query's planning critical path (tens of ms typical, more on high-latency object stores/metastores)", + "confidence": "high", + "lens": "scan-planning-loop", + "dupes": [ + "streamingSplitEstimate re-reads the manifest list and rebuilds the scan that planScan/streamSplits immediately re-derive on a fresh Table [chain-redundancy]" + ], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Re-derived every hop independently. (1) Heaviness holds: streamingSplitEstimate:410 resolveTable -> IcebergCatalogOps.CatalogBackedIcebergCatalogOps.loadTable (IcebergCatalogOps.java:340-342) is a raw catalog.loadTable — remote per the SDK cost model; there is no CachingCatalog and no table-object cache on the scan path (the scan provider is wired with raw newCatalogBackedOps at IcebergConnector.java:589-591/:464-486; the IcebergLatestSnapshotCache goes only to getMetadata at :214). Then :424-425 snapshot.dataManifests(table.io()) reads the manifest-list file remotely on the freshly loaded Table (SDK io.manifest.cache-enabled defaults OFF: IcebergCatalogFactory.java:68,:117-133). The summing itself (:426-431) and getMatchingManifest (:1809-1823) are in-memory, as claimed. (2) Multiplicity holds exactly as claimed: FileQueryScanNode.createScanRangeLocations:376 -> isBatchMode (memoized once per query via isBatchModeCache, PluginDrivenScanNode.java:1385-1390) -> computeBatchMode:1414 runs the estimate on every sloted scan regardless of outcome; then BOTH terminal paths redo the work on their own fresh Table with zero handoff — sync: getSplits:1242 planScan -> planScanInternal:562 second loadTable + :627 planFiles (re-reads the same manifest list + all manifests); batch: startStreamingSplit:1633-1634 streamSplits -> provider :452 second loadTable + :463 scan.planFiles(). resolveTable (:1981-1993) has no memo, and resolveScanProvider (:224-226) builds a FRESH provider per call (IcebergConnector.java:584 comment), so even a provider-field Table memo would not survive the estimate->plan handoff. The design comment at PluginDrivenScanNode.java:1614-1619 explicitly states the estimate ran pre-pin on the current snapshot, decoupled from the pinned planning pass — deliberate for correctness but confirming nothing is reused. This is variant B (single-chain repeated remote fetch of the same info: loadTable x2, manifest-list read x2 per query) per the problem-class doc §3. Minor gates that skip the cost (noted, do not refute): system tables and enable_external_table_batch_mode=false (:407, default true), no output slots (:1412), TABLESAMPLE on a sample-supporting connector (:1404-1406). A normal iceberg data query pays the redundant loadTable + manifest-list read on the synchronous planning critical path every time.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 376, + "note": "createScanRangeLocations calls isBatchMode() on every query's planning path; else-branch :422 getSplits is the sync path" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1385, + "note": "isBatchMode memoized via isBatchModeCache (:1386-1389) — estimate runs exactly once per query, k=1 as claimed" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1414, + "note": "computeBatchMode invokes scanProvider.streamingSplitEstimate whenever the scan has slots (:1412), regardless of whether streaming is chosen" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 410, + "note": "streamingSplitEstimate: resolveTable = first loadTable of the query's scan planning; gates at :407 (isSystemTable, ENABLE_EXTERNAL_TABLE_BATCH_MODE default true) pass for normal queries" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 424, + "note": "snapshot.dataManifests(table.io()) reads the manifest-list file remotely on the freshly loaded Table; the summing loop :426-431 and getMatchingManifest :1809-1823 are in-memory" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981, + "note": "resolveTable has no memoization — every call is ops.loadTable (:1985/:1989), auth-wrapped" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "CatalogBackedIcebergCatalogOps.loadTable = raw catalog.loadTable — remote IO per SDK cost model; no CachingCatalog wrap anywhere in the connector" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 589, + "note": "getScanPlanProvider wires the raw newCatalogBackedOps (fresh provider per call, :584 comment); the IcebergLatestSnapshotCache goes only to getMetadata (:214), not the scan path" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 562, + "note": "sync path planScanInternal: second resolveTable (loadTable), then :627 planFileScanTask -> planFiles re-reads the same manifest list + manifests; reached via PluginDrivenScanNode.getSplits:1242" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 452, + "note": "batch path streamSplits: second resolveTable (loadTable) + :463 scan.planFiles(); reached via PluginDrivenScanNode.startStreamingSplit:1633-1634" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1615, + "note": "design comment: the streamingSplitEstimate gate ran pre-pin on the current snapshot, decoupled from the pinned planning pass — confirms nothing (Table/snapshot/manifest list) is shared by design" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE=false — SDK io.manifest.cache-enabled derivation is default-off, so the duplicated manifest-list read is a real remote read by default" + } + ], + "corrected_multiplicity": "k-times-per-query with k=1 extra (loadTable + manifest-list read), exactly as claimed — memoized via isBatchModeCache so the estimate never repeats; total per normal iceberg query: loadTable x2 and manifest-list read x2 (estimate + planScan/streamSplits), both on the synchronous planning critical path. Skipped only for: system tables, enable_external_table_batch_mode=false, no-output-slot scans, and TABLESAMPLE on a sample-supporting connector.", + "fix_direction": "Share the resolved Table (and ideally the snapshot/manifest-list) between the batch-mode estimate and the immediately-following planScan/streamSplits. Because resolveScanProvider builds a fresh provider per call, a provider-instance memo is insufficient — the hoist must live at the connector level (e.g. a short-TTL per-query Table memo keyed by db.table+queryId inside IcebergConnector, analogous to legacy fe-core IcebergMetadataCache, honoring the existing meta.cache.iceberg.table.ttl-second knob) or as an explicit handoff (streamingSplitEstimate stashes the loaded Table/scan keyed by queryId, planScan/streamSplits consume it; the rewritableDeleteStash already demonstrates this queryId-stash pattern in the same class). The estimate-vs-pin snapshot divergence documented at PluginDrivenScanNode.java:1614-1619 must be preserved (the planning pass may pin a different snapshot, so reuse the Table object, not the estimate's snapshot resolution).", + "impact_estimate": "Every normal iceberg query pays +1 catalog.loadTable (metastore RPC or REST round-trip + metadata.json fetch) and +1 manifest-list file read on the synchronous planning critical path, on top of the identical work the planning pass then redoes. Constant-factor (~2x the table-load/manifest-list portion of planning latency), not split-amplified: typically tens of ms per query, more on high-latency REST catalogs / object stores or Kerberos-wrapped HMS; multiplied by QPS across concurrent queries and adding metastore load.", + "mitigation_found": "isBatchModeCache (PluginDrivenScanNode.java:1385-1390) caps the estimate at once per query — the claim already accounts for this. No other mitigation on the path: no CachingCatalog, no table-object cache on the scan path (IcebergLatestSnapshotCache serves only getMetadata), resolveTable is unmemoized, and the SDK manifest cache (io.manifest.cache-enabled) is default-off — enabling meta.cache.iceberg.manifest.* would blunt the manifest-list re-read but not the duplicate loadTable." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The candidate's chain is real and no layer shares the heavy prerequisites between the batch-mode estimate and the actual split planning. Verified chain: FileQueryScanNode.createScanRangeLocations branches on isBatchMode() (FileQueryScanNode.java:376); PluginDrivenScanNode.isBatchMode memoizes via isBatchModeCache (PluginDrivenScanNode.java:1385-1390) so computeBatchMode runs once per query; computeBatchMode calls streamingSplitEstimate whenever the scan has output slots (PluginDrivenScanNode.java:1412-1415). streamingSplitEstimate does resolveTable (IcebergScanPlanProvider.java:410) -> catalogOpsResolver -> CatalogBackedIcebergCatalogOps.loadTable -> catalog.loadTable (IcebergCatalogOps.java:340-341), a raw SDK remote load (metastore RPC/REST call + metadata.json fetch), then reads the snapshot's manifest-list via snapshot.dataManifests(table.io()) (IcebergScanPlanProvider.java:424-425). Whatever the estimate returns, the subsequent path redoes both remote ops on its OWN freshly loaded Table: sync path planScanInternal calls resolveTable again (line 562) and planFileScanTask -> planFiles (line 627) re-reads the same manifest-list on a new Snapshot object; batch path streamSplits calls resolveTable (line 452) and scan.planFiles() (line 463). Mitigation hunt results: (1) isBatchModeCache only caps the estimate at k=1, it shares nothing with planScan; (2) IcebergLatestSnapshotCache caches only (snapshotId, schemaId) pins for beginQuerySnapshot (IcebergLatestSnapshotCache.java:30-52), never Table objects, and resolveTable never consults it; (3) IcebergManifestCache caches per-manifest data/delete entries for the sync planning path only — not manifest-LIST reads and not table loads — and streamSplits explicitly bypasses it (comment at IcebergScanPlanProvider.java:444-446); (4) the SDK CachingCatalog is used nowhere (zero grep hits in fe-connector-iceberg/fe-connector-metastore-iceberg), so every catalog.loadTable is a fresh remote load; (5) the io.manifest.cache-enabled derivation (IcebergCatalogFactory.java:115-131) feeds the SDK's manifest-FILE content cache (ManifestFiles.read path) — the manifest-list read in BaseSnapshot.cacheManifests goes through plain io.newInputFile and is not covered, nor is metadata.json; (6) IcebergScanPlanProvider has no per-query Table memo — resolveTable (lines 1981-1993) is a plain pass-through, and getScanNodeProperties even performs a THIRD independent loadTable per query (line 1300); the code's own comment at lines 1412-1416 shows the authors dodge redundant loadTables within one method but have no cross-method sharing. Gates: the extra cost is skipped only for system tables, scans with no output slots, or enable_external_table_batch_mode=false — and that session variable defaults to TRUE (SessionVariable.java:3041), so the default configuration pays it on every normal iceberg SELECT. This is variant B (single-chain redundancy) per the problem-class doc: the estimate itself is cheap by design, but its two remote prerequisites duplicate work the immediately following planScan/streamSplits repeats.", + "corrected_multiplicity": "per-query-constant: exactly +1 redundant loadTable (remote metastore RPC/REST + metadata.json read) and +1 redundant manifest-list file read per iceberg data-table query with output slots (isBatchModeCache caps the estimate at k=1; the candidate's multiplicity statement is accurate). Note the same query also pays a third independent loadTable in getScanNodeProperties (IcebergScanPlanProvider.java:1300) — adjacent to but outside this finding's scope.", + "mitigation_found": "Partial only. isBatchModeCache (PluginDrivenScanNode.java:1386-1389) prevents the estimate from running more than once per query, and the enable_external_table_batch_mode=false session variable would skip it entirely — but that variable defaults to true (SessionVariable.java:3041). No mechanism shares the Table/Snapshot/manifest-list between the estimate and planScan/streamSplits: no SDK CachingCatalog, no Table-object cache (IcebergLatestSnapshotCache holds only snapshotId+schemaId pins), IcebergManifestCache covers per-manifest entries on the sync path only (not manifest-list reads, not loadTable), and the io.manifest content cache covers manifest files, not the manifest-list or metadata.json.", + "impact_estimate": "O(1)-per-query x remote-IO (1 extra catalog.loadTable = metastore/REST round trip + metadata.json object-store read, plus 1 extra manifest-list avro read) x broad trigger (all iceberg data tables, every SELECT with output slots, default session config; both the batch and sync outcomes pay it). Typically tens of ms added to the planning critical path per query, scaling with metastore/object-store latency and QPS; worst on high-latency REST catalogs and S3-class stores.", + "fix_direction": "pass-down / memoize: share one resolved Table across the estimate and the subsequent plan call within a query — either a per-query (queryId, db.table)-keyed Table memo inside IcebergScanPlanProvider consulted by resolveTable, or an SPI-level handoff where streamingSplitEstimate's resolved Table/scan context is carried into planScan/streamSplits (and ideally getScanNodeProperties). Sharing the Table instance removes BOTH duplicated remote costs automatically: the second loadTable disappears, and planFiles on the same BaseSnapshot object reuses the manifest list cached by the estimate's dataManifests call.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1386, + "note": "isBatchMode memoized via isBatchModeCache — computeBatchMode (and thus streamingSplitEstimate) runs exactly once per query; confirms k=1 multiplicity but shares no Table with the later plan call" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1414, + "note": "computeBatchMode invokes scanProvider.streamingSplitEstimate whenever the scan has output slots, regardless of whether streaming is ultimately chosen" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 376, + "note": "createScanRangeLocations branches on isBatchMode(); false falls to getSplits (line 422) -> planScan, true goes to the SplitAssignment/streamSplits path — both redo the table load" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 410, + "note": "streamingSplitEstimate: resolveTable = fresh remote loadTable; gate at lines 407-409 only skips for system tables or batch-mode session var off (default on)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 424, + "note": "snapshot.dataManifests(table.io()) — manifest-list file read on the estimate's fresh Snapshot object; not covered by IcebergManifestCache (per-manifest entries) nor the SDK io.manifest content cache (manifest files only)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 562, + "note": "planScanInternal calls resolveTable again (second loadTable) and planFileScanTask at line 627 re-reads the same manifest list via planFiles on a new Snapshot" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 452, + "note": "batch path streamSplits also calls resolveTable (second loadTable) then scan.planFiles() at line 463 (manifest-list re-read); comment at 444-446 says this path deliberately bypasses the manifest cache" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981, + "note": "resolveTable is a plain pass-through to catalogOps.loadTable with no per-query memo; getScanNodeProperties line 1300 does a third independent loadTable, and the comment at 1412-1416 confirms the authors count loadTable round-trips but have no cross-method Table sharing" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "CatalogBackedIcebergCatalogOps.loadTable delegates straight to catalog.loadTable(identifier) — raw SDK catalog from IcebergConnector.getOrCreateCatalog, never wrapped in CachingCatalog (zero hits in both iceberg modules)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java", + "line": 57, + "note": "the only table-level cache stores just (snapshotId, schemaId) pins for beginQuerySnapshot — not Table objects; irrelevant to resolveTable's loadTable cost" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java", + "line": 3041, + "note": "enableExternalTableBatchMode defaults to true — the estimate (and its redundant remote ops) runs on every normal iceberg query in default configuration" + } + ] + } + }, + { + "title": "Streaming split pump feeds SplitAssignment one split at a time — per-split computeScanRangeAssignment rebuilds the backend candidate set and takes the assign lock per range", + "variant": "A-loop-amplification", + "heavy_op": "FederationBackendPolicy.computeScanRangeAssignment on a singleton list: full backend-list copy (backendMap.values() flatten), new ResettableRandomizedIterator, new ArrayListMultimap, shuffle — plus a synchronized(assignLock) round-trip and a per-range BlockingQueue offer, all repeated per split", + "multiplicity": "per-split, on the streaming batch path only — which by construction activates when matched file count >= num_files_in_batch_mode (default 1024) up to the million-file scans this path exists to protect: pump loop at PluginDrivenScanNode.startStreamingSplit:1638-1642 wraps EACH range in a 1-element list", + "entry_chain": "PluginDrivenScanNode.startStreamingSplit (fe-core/.../PluginDrivenScanNode.java:1638-1642, one.add(new PluginDrivenSplit(source.next())); splitAssignment.addToQueue(one)) -> SplitAssignment.addToQueue (fe-core/.../datasource/split/SplitAssignment.java:143-156, synchronized + computeScanRangeAssignment per call) -> FederationBackendPolicy.computeScanRangeAssignment (fe-core/.../datasource/scan/FederationBackendPolicy.java:225-235 backends copy + iterator rebuild) -> SplitAssignment.appendBatch:109-129 (per-backend queue offer of a 1-element collection, and splitToScanRange per split)", + "files": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1638 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/split/SplitAssignment.java", + "line": 153 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java", + "line": 232 + } + ], + "why_heavy": "Each call is O(#backends) allocation + collection copies + lock acquisition; loop-invariant state (the flattened backend list, the randomized iterator) is rebuilt from scratch every iteration because the batching that would amortize it is set to 1. Individually micro, but this is the designated path for 10^5–10^6-split scans, so aggregate cost is O(N x #backends) CPU + N lock/queue round-trips serialized on the single pump thread, directly extending time-to-first-split-batch and total split-generation time. Micro-batching the pump (e.g. drain 64-256 ranges per addToQueue) hoists all of it; backpressure semantics (needMoreSplit + bounded queue) are unaffected", + "gate": "streaming batch flavor only (streamingSplitEstimate >= num_files_in_batch_mode, enable_external_table_batch_mode on, format-version < 3, non-sys, no servable COUNT pushdown)", + "est_impact": "at 10^5-10^6 splits x ~100 BEs: order 10^7-10^8 redundant ops + 10^5-10^6 lock/queue round-trips on the pump thread — roughly seconds of added split-generation latency on exactly the largest scans", + "confidence": "medium", + "lens": "scan-planning-loop", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Independently re-derived every hop. (1) Multiplicity per-split holds: PluginDrivenScanNode.startStreamingSplit's pump loop (lines 1638-1642) wraps EACH range from the lazy ConnectorSplitSource in a 1-element ArrayList and calls splitAssignment.addToQueue per range; startSplit dispatches to it when streamingBatch (1498-1502), which computeBatchMode sets when the connector estimate >= 0 (1412-1420). Iceberg's IcebergScanPlanProvider.streamingSplitEstimate (404-437) returns the matched-file count only when it reaches num_files_in_batch_mode (DEFAULT_NUM_FILES_IN_BATCH_MODE=1024, line 142) with enable_external_table_batch_mode on (default true, line 407), snapshot present, non-sys, format-version < 3, no servable COUNT pushdown — the gate is exactly as claimed, and by construction N >= 1024 up to the million-file scans the path exists for. The pump runs to full enumeration (backpressure via queue.offer 100ms retries only slows it). (2) Heaviness-as-amplification holds and is UNDERSTATED: per addToQueue call, SplitAssignment takes assignLock (143-156) and calls FederationBackendPolicy.computeScanRangeAssignment on the singleton (line 153), which per call allocates an ArrayListMultimap (226), a new Random + no-op shuffle (228), re-flattens backendMap into a fresh list (231-234) even though the policy's own `backends` field already holds exactly that list (init, line 185), and constructs a ResettableRandomizedIterator whose ctor copies the whole backend collection (ResettableRandomizedIterator.java:36-38) despite the default ROUND_ROBIN strategy (FederationBackendPolicy:148-150) never using it. Additionally — missed by the finding — enableSplitsRedistribution defaults to true (line 92) and its setter is @VisibleForTesting with zero production callers, so equateDistribution (320-343) runs per split: a third backend flatten, Collections.sort O(B log B), and TWO IndexedPriorityQueue builds over all backends per call; this dominates per-call cost. appendBatch (109-129) then offers a singleton collection per call. No hoist/memoize exists at any layer; micro-batching the pump would amortize all of it while preserving needMoreSplit backpressure. Caveats: (a) no remote IO — each call is O(B log B) CPU + ~8 allocations, single-digit microseconds at ~100 BEs; aggregate ~0.2-10 s serialized on the single pump thread at 10^5-10^6 splits, consistent with the finding's own 'individually micro, aggregate seconds' framing; relative to inherent per-split work (buildRangeForTask + splitToScanRange) the amplification is a ~1.5-3x pump-CPU multiplier, not orders of magnitude, and it overlaps BE execution. (b) This is a faithful port of legacy IcebergScanNode.doStartSplit one-at-a-time pumping into unchanged upstream fe-core machinery (comment at 1635-1637) — an inherited engine pattern, not an SPI-migration regression — which does not refute the class match (variant A/D: loop-invariant state rebuilt per iteration, no layer hoists it).", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1638, + "note": "Pump loop: while(needMoreSplit && hasNext) { one=new ArrayList<>(1); one.add(new PluginDrivenSplit(source.next())); splitAssignment.addToQueue(one); } — one split per addToQueue, per-split multiplicity confirmed" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1498, + "note": "startSplit dispatches to startStreamingSplit when streamingBatch; streamingBatch set at 1412-1420 when connector streamingSplitEstimate >= 0" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 437, + "note": "return fileCount >= threshold ? fileCount : -1 — streaming activates exactly when matched files >= num_files_in_batch_mode (default 1024, line 142); gates at 407-421 (batch-mode session var default true, non-sys, snapshot, format<3, count-pushdown) match the claimed gate" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/split/SplitAssignment.java", + "line": 153, + "note": "addToQueue: synchronized(assignLock) at 148 + backendPolicy.computeScanRangeAssignment(splits) per call at 153; appendBatch 109-129 offers a singleton collection per call with 100ms-retry offer loop" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java", + "line": 232, + "note": "Per call: ArrayListMultimap.create (226), new Random+shuffle on 1 element (228), backendMap flatten into fresh list (231-234) duplicating the existing `backends` field populated at init (185), new ResettableRandomizedIterator (235) unused under default ROUND_ROBIN (148-150)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java", + "line": 307, + "note": "enableSplitsRedistribution defaults true (92), setter @VisibleForTesting with no production caller (repo-wide grep) — so equateDistribution runs per singleton call: third flatten + Collections.sort O(B log B) (329) + two IndexedPriorityQueue builds over all backends (335-343); dominates per-call cost, missed by the original finding" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/common/ResettableRandomizedIterator.java", + "line": 37, + "note": "Ctor copies the whole backend collection (new ArrayList<>(elements)) — O(#backends) allocation per split, dead weight under default ROUND_ROBIN strategy" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1635, + "note": "Comment: 'pump them one at a time, exactly like legacy doStartSplit' — inherited legacy pattern in unchanged upstream fe-core machinery, not an SPI-migration regression (context, not refutation)" + } + ], + "corrected_multiplicity": "per-split (N = matched file count post size-slicing, >= 1024 by gate construction, up to 10^5-10^6 on the scans this path protects), serialized on the single pump thread; each split additionally pays the equateDistribution heap-build over all backends, which the original finding did not count", + "mitigation_found": "None on this chain. isBatchModeCache memoizes only the batch decision; nothing batches between source.next() and addToQueue; FederationBackendPolicy rebuilds loop-invariant state per call (its own `backends` field already holds the flattened list but is not reused); enableSplitsRedistribution cannot be disabled in production. Backpressure (needMoreSplit + bounded 10000-entry queues) limits heap, not CPU amplification.", + "fix_direction": "Micro-batch the pump: drain e.g. 64-256 ranges from the ConnectorSplitSource per addToQueue call (checking needMoreSplit per batch) — preserves backpressure semantics while amortizing lock, multimap, backend-flatten, iterator, and equateDistribution costs by the batch factor; batching also makes the fixed-seed shuffle and redistribution logic actually do their intended balancing work (both are no-ops on singletons). Secondarily, inside FederationBackendPolicy: reuse the `backends` field instead of re-flattening, construct ResettableRandomizedIterator only for the RANDOM strategy, and skip/incrementalize equateDistribution for tiny batches. Engine-side (fe-core) change; note SplitAssignment/FederationBackendPolicy are upstream-shared code, so consider upstreaming.", + "impact_estimate": "Real but bounded: ~2-10 us of loop-invariant CPU + 1 uncontended lock + 1 queue-offer per split at ~100 BEs (equateDistribution dominating), i.e. roughly 0.2-1 s at 10^5 splits and 2-10 s at 10^6 splits of added serialized pump-thread time — a ~1.5-3x multiplier on inherent per-split pump work, extending time-to-split-delivery on exactly the largest streaming scans; not orders-of-magnitude, no remote IO amplification." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "All cited mechanics verified by reading the code. The streaming pump (PluginDrivenScanNode.startStreamingSplit:1638-1641) hardcodes a 1-element list per SplitAssignment.addToQueue call; each call takes assignLock and runs FederationBackendPolicy.computeScanRangeAssignment (SplitAssignment.java:148-153), which unconditionally rebuilds loop-invariant state per call: new ArrayListMultimap (line 226), Collections.shuffle with a new Random (228), full backendMap flatten into a fresh ArrayList (231-234), and a new ResettableRandomizedIterator whose ctor copies the list again (235; ResettableRandomizedIterator.java:36). The finding actually UNDERSTATES the per-call cost: enableSplitsRedistribution defaults to true (FederationBackendPolicy.java:92) and its only setter is @VisibleForTesting with zero production callers (grep-verified), so equateDistribution (307-308, 325-343) additionally re-flattens all backends, sorts them O(B log B), and builds TWO IndexedPriorityQueues over all backends on EVERY per-split call before its early-return variance check. Mitigation hunt results: (1) the static consistentHashCache (lines 95-106, used at init:200) only amortizes hash-ring construction, not the per-call rebuilds; (2) batching exists on both sibling paths — the partition-batch flavor passes whole partition batches to addToQueue (PluginDrivenScanNode.java:1565-1570) and the synchronous path assigns all splits in ONE computeScanRangeAssignment call (FileQueryScanNode.java:431) — but not on the streaming pump; (3) no config can raise the pump batch size; (4) consumer-side draining does not help producer-side per-call cost. Gate verified: iceberg streaming activates when manifest-metadata matched-file count >= num_files_in_batch_mode (IcebergScanPlanProvider.java:404-437, threshold default 1024 via SessionVariable.java:2578; enableExternalTableBatchMode=true default at 3041; excluded: sys tables, empty table, v>=3, servable COUNT pushdown). Honest caveats that temper severity: the amplified cost is local CPU + allocation only (no remote IO — the heavy manifest IO in the source is inherent and streamed once); per-call cost is ~1-5us at ~100 BEs, so aggregate is roughly 0.1s at 10^5 splits to seconds at 10^6 splits plus ~10 transient allocations per split of GC pressure, all serialized on the single pump thread; and the one-at-a-time pump is a faithful port of upstream legacy IcebergScanNode.doStartSplit (comment at 1635-1637), i.e. parity with upstream Doris, not an SPI-migration regression. It still meets the problem class: loop-invariant work (variant D inside variant A) at per-split multiplicity with no hoist/memoize at any layer in the default configuration.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1638, + "note": "Pump loop: while(needMoreSplit && hasNext) { one = new ArrayList<>(1); one.add(new PluginDrivenSplit(source.next())); splitAssignment.addToQueue(one); } — 1 split per addToQueue call; comment at 1635-1637 says 'pump them one at a time, exactly like legacy doStartSplit'" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/split/SplitAssignment.java", + "line": 153, + "note": "addToQueue (143-156): synchronized(assignLock) + backendPolicy.computeScanRangeAssignment(splits) per call; appendBatch (109-129) then does splitToScanRange + computeIfAbsent + bounded queue.offer of a 1-element collection per call" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java", + "line": 232, + "note": "computeScanRangeAssignment (225-311) per call: new ArrayListMultimap (226), shuffle+new Random (228), backendMap flatten into fresh ArrayList (231-234), new ResettableRandomizedIterator (235) — all loop-invariant, rebuilt every call" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/common/ResettableRandomizedIterator.java", + "line": 36, + "note": "Constructor copies the whole backend collection: this.list = new ArrayList<>(elements) — O(#backends) per computeScanRangeAssignment call" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java", + "line": 92, + "note": "enableSplitsRedistribution = true by default; setter (212-215) is @VisibleForTesting with no production caller (grep over fe/ main sources: only FederationBackendPolicy itself) — so equateDistribution runs per call" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java", + "line": 326, + "note": "equateDistribution (320-343): per call re-flattens backendMap, Collections.sort O(B log B), builds TWO IndexedPriorityQueues over all backends — per-split cost the finder did not even count; variance early-return (358-360) only after this setup" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FederationBackendPolicy.java", + "line": 200, + "note": "PARTIAL mitigation found: static consistentHashCache (95-106) amortizes hash-ring construction, but only at init() — does not touch the per-call rebuilds" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 431, + "note": "Contrast/mitigation on sibling path: non-batch path calls computeScanRangeAssignment ONCE with the full split list — the amortization exists everywhere except the streaming pump" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1570, + "note": "Contrast: partition-batch flavor batches — splitAssignment.addToQueue(batchSplits) with a whole partition batch, amortizing the assignment machinery" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 437, + "note": "Gate verified: streamingSplitEstimate returns fileCount only when >= num_files_in_batch_mode threshold (default 1024; DEFAULT_NUM_FILES_IN_BATCH_MODE=1024L at line 142); excluded: sys tables, batch mode off, empty table, format-version>=3, servable COUNT pushdown (404-421)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java", + "line": 2578, + "note": "numFilesInBatchMode = 1024 default; enableExternalTableBatchMode = true default (line 3041) — streaming flavor is ON by default for iceberg scans matching >= 1024 files" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1416, + "note": "computeBatchMode: streaming flavor activates whenever connector estimate >= 0 (1412-1420), i.e. exactly the >= 1024-matched-files iceberg gate; startSplit dispatches to startStreamingSplit when streamingBatch (1498-1501)" + } + ], + "corrected_multiplicity": "per-split (O(splits)) on the streaming batch path only — activates at matched-file count >= 1024 (default) and is the designated path for 10^5-10^6-split scans; each split pays O(#backends) copies + O(B log B) equateDistribution setup + 1 lock round-trip + 1 bounded-queue offer", + "mitigation_found": "No mitigation prevents the repeated cost in default config. Partial/adjacent mechanisms found: (1) static consistentHashCache amortizes only hash-ring construction at init, not per-call rebuilds; (2) batching exists on BOTH sibling paths (partition-batch flavor batches per partition batch at PluginDrivenScanNode:1570; sync path assigns all splits in one call at FileQueryScanNode:431) but not the streaming pump; (3) enableSplitsRedistribution=false would remove the largest per-call term (equateDistribution) but its setter is @VisibleForTesting with zero production callers — off-by-default is unreachable. Also note: the one-at-a-time pump is a deliberate port of upstream legacy IcebergScanNode.doStartSplit, so this is parity with upstream, not an SPI-migration regression.", + "fix_direction": "hoist + micro-batch: (a) micro-batch the pump — drain k (e.g. 64-256) ranges from the source per addToQueue call; backpressure is preserved since appendBatch already offers with timeout under needMoreSplit and the per-backend queue is bounded at 10000; (b) hoist loop-invariant state in FederationBackendPolicy — precompute the flattened backend list and ResettableRandomizedIterator once at init, and make equateDistribution's allNodes/sort/heap setup lazy or incremental instead of per-call. (a) alone amortizes everything by k and is the minimal surgical fix.", + "impact_estimate": "O(splits) x local-cpu (allocation + O(#backends log #backends) per call, NO remote IO) x iceberg streaming-batch scans only (matched files >= num_files_in_batch_mode=1024 default, enable_external_table_batch_mode=true default, non-sys, format-version < 3, no servable COUNT pushdown). At ~100 BEs: ~1-5us + ~10-15 transient allocations per split, serialized on the single pump thread — negligible at the 1024-file threshold, roughly 0.1-0.5s at 10^5 splits, and low single-digit seconds plus ~10^7 transient objects of GC pressure at 10^6 splits, added to split-generation time on exactly the largest scans. Materially smaller than the remote-IO findings of this class, and identical to upstream legacy behavior." + } + }, + { + "title": "One SELECT planning pass performs ~7 independent remote catalog.loadTable() of the same table (no table-object cache at any layer)", + "variant": "B-chain-redundancy", + "heavy_op": "IcebergCatalogOps.CatalogBackedIcebergCatalogOps.loadTable -> raw iceberg SDK catalog.loadTable() (REST HTTP loadTable / HMS getTable RPC + metadata.json fetch per call; no CachingCatalog wrap, no connector-side Table cache)", + "multiplicity": "k-times-per-query (k≈7 sync path, 9 streaming): (1) getOrLoadPropertiesResult->buildColumnHandles PluginDrivenScanNode.java:1780->1856; (2) getScanNodeProperties->resolveTable IcebergScanPlanProvider.java:1300; (3) computeBatchMode->streamingSplitEstimate->resolveTable IcebergScanPlanProvider.java:410; (4) getSplits->buildColumnHandles PluginDrivenScanNode.java:1202; (5) planScan->resolveTable IcebergScanPlanProvider.java:562; (6) materializeLatest->getMvccPartitionView IcebergConnectorMetadata.java:1453; (7) listPartitions IcebergConnectorMetadata.java:1507; streaming adds startStreamingSplit->buildColumnHandles:1611 and streamSplits->resolveTable IcebergScanPlanProvider.java:452. Plus 2 tableExists RPCs (PluginDrivenScanNode.java:207, PluginDrivenMvccExternalTable.java:145 -> IcebergConnectorMetadata.java:324)", + "entry_chain": "FileQueryScanNode.createScanRangeLocations (FileQueryScanNode.java:325,376,422) -> PluginDrivenScanNode.getFileFormatType:528 -> getOrLoadScanNodeProperties:1815 -> getOrLoadPropertiesResult:1776 -> [buildColumnHandles:1780 -> IcebergConnectorMetadata.getColumnHandles:587 -> loadTable:540] + [getScanNodePropertiesResult (ConnectorScanPlanProvider.java:455) -> IcebergScanPlanProvider.getScanNodeProperties:1294 -> resolveTable:1300 -> :1981-1989]; then isBatchMode -> computeBatchMode:1414 -> streamingSplitEstimate -> resolveTable (IcebergScanPlanProvider.java:410); then getSplits:1156 -> buildColumnHandles:1202 + planScan:1242 -> planScanInternal -> resolveTable (IcebergScanPlanProvider.java:562); each resolveTable/loadTable -> IcebergCatalogOps.java:340-341 catalog.loadTable()", + "files": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1856 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 540 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340 + } + ], + "why_heavy": "Every loadTable is a full remote metadata round trip: REST = one HTTP loadTable (server-side planning of table metadata + optional vended creds); HMS/Glue = metastore getTable RPC plus reading the current metadata.json from object storage. Verified NO caching layer exists: IcebergCatalogFactory never wraps with CachingCatalog; IcebergConnector.getScanPlanProvider builds a fresh provider per call (IcebergConnector.java:583-592); IcebergLatestSnapshotCache stores only (snapshotId, schemaId) (IcebergLatestSnapshotCache.java:57-65), never a Table object. Legacy fe-core served the Table from IcebergMetadataCache once per catalog.", + "est_impact": "~6 redundant remote metadata loads per query per iceberg table: roughly +0.1s to +3s planning latency per query depending on metastore RTT, and ~7x metastore/object-store QPS amplification vs a single cached load. Hits every query on every iceberg catalog flavor.", + "confidence": "high", + "lens": "chain-redundancy", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Both heaviness and multiplicity hold as claimed. Heaviness: IcebergCatalogOps.CatalogBackedIcebergCatalogOps.loadTable (line 340-342) is a raw SDK catalog.loadTable delegation — HMS/Glue = getTable RPC + metadata.json object-store read, REST = HTTP loadTable; grep confirms zero CachingCatalog usage in the connector; IcebergScanPlanProvider.resolveTable (1981-1993) and IcebergConnectorMetadata.loadTable (540-547) have no memoization; IcebergLatestSnapshotCache stores only (snapshotId, schemaId) ids (lines 57-65), never a Table; IcebergConnector.getScanPlanProvider builds a fresh provider per call (583-592). Multiplicity: I independently traced one plain SELECT of a partitioned iceberg table and count 6-7 independent remote loadTable calls plus 2 remote tableExists calls: (a) MVCC query-begin pin via StatementContext.loadSnapshots → materializeLatest → getTableHandle/tableExists (RPC), getMvccPartitionView loadTable (IcebergConnectorMetadata:1453), and for partitioned non-RANGE tables listPartitions loadTable (:1507); (b) PluginDrivenScanNode.create:207 → getTableHandle/tableExists (RPC) again; (c) getOrLoadPropertiesResult's single execution does TWO loads — buildColumnHandles→getColumnHandles:587→loadTable:540 and getScanNodeProperties:1300→resolveTable; (d) isBatchMode→streamingSplitEstimate→resolveTable (IcebergScanPlanProvider:410, gated on enable_external_table_batch_mode default-true); (e) getSplits does a SECOND un-memoized buildColumnHandles (PluginDrivenScanNode:1202) plus planScan→planScanInternal:562→resolveTable. beginQuerySnapshot adds one more loadTable on snapshot-cache TTL miss/disable. The candidate's claim that per-split getFileFormatType is memoized is also correct (cachedPropertiesResult, PluginDrivenScanNode:1776-1809), so this is k-times-per-query redundancy, not per-split — exactly as claimed. Only one loadTable (or arguably zero, given the schema cache) is semantically needed per planning pass; the rest re-fetch identical table metadata within milliseconds of each other with no hoist at any layer.", + "corrected_multiplicity": "k-times-per-query: sync path = 6-7 remote catalog.loadTable + 2 remote tableExists per SELECT per partitioned iceberg table (5-6 loadTable for unpartitioned; +1 loadTable when the latest-snapshot cache TTL misses or is disabled); streaming path swaps getSplits' planScan for streamSplits→resolveTable (IcebergScanPlanProvider:452) plus another buildColumnHandles, giving a similar or higher count. The claim's k≈7 is accurate.", + "mitigation_found": "Three partial mitigations exist, none caching the Table object: (1) PluginDrivenScanNode.cachedPropertiesResult (PluginDrivenScanNode.java:1776-1809) memoizes the scan-node properties so the per-split getFileFormatType at FileQueryScanNode.java:485 does NOT re-plan — the canonical per-split bug is absent; (2) IcebergLatestSnapshotCache TTL-caches only (snapshotId, schemaId) for beginQuerySnapshot (IcebergLatestSnapshotCache.java:57-65), so the pin is usually cheap but every other site still re-loads; (3) IcebergManifestCache reduces manifest IO inside planFiles but not loadTable itself. No CachingCatalog wrap, no connector-level Table cache, no per-query Table memoization anywhere.", + "fix_direction": "Memoize the resolved Table per (table, pinned-snapshot) at a layer both IcebergConnectorMetadata and IcebergScanPlanProvider share. Natural seam: a per-catalog Table cache on the long-lived IcebergConnector (mirroring legacy fe-core IcebergMetadataCache), keyed by TableIdentifier, honoring the existing meta.cache.iceberg.table.ttl-second spec and the same REFRESH TABLE/DATABASE/CATALOG invalidation hooks IcebergLatestSnapshotCache already has — or equivalently wrap the SDK catalog with CachingCatalog plus explicit invalidation. Cheaper incremental step: a per-query (queryId-keyed or scan-node-scoped) Table memo threaded through resolveTable/loadTable, plus reusing the buildColumnHandles result between getOrLoadPropertiesResult and getSplits in PluginDrivenScanNode. Also fold the duplicate getTableHandle/tableExists probes (scan-node create re-checks existence the MVCC pin already established).", + "impact_estimate": "Per SELECT per iceberg table: ~5-6 redundant remote metadata round trips beyond the 1 needed (each = HMS getTable RPC + metadata.json fetch from object storage, or one REST loadTable HTTP call with optional cred vending), plus 1 redundant tableExists RPC. At 20-500ms per round trip this adds roughly 0.1s-3s of serialized planning latency per query and ~6-7x metastore/object-store metadata QPS vs a single cached load; hits every query on every iceberg catalog flavor (REST/HMS/Glue/hadoop), worst on high-RTT REST or throttled Glue/S3.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "loadTable = raw catalog.loadTable(toTableIdentifier(...)), no cache; tableExists at line 303-305 likewise raw" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981, + "note": "resolveTable: every call resolves ops and calls ops.loadTable (1985/1989); no memoization field" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 410, + "note": "streamingSplitEstimate calls resolveTable; also streamSplits:452, planScanInternal:562, getScanNodeProperties:1300" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 540, + "note": "loadTable wrapper straight to catalogOps.loadTable; getColumnHandles:587 loads per call; getTableHandle:324 remote tableExists; getMvccPartitionView:1453 and listPartitions:1507 each loadTable; beginQuerySnapshot:1540-1545 loads only on snapshot-cache miss" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java", + "line": 57, + "note": "CachedSnapshot holds only snapshotId+schemaId — no Table object cached" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 583, + "note": "getScanPlanProvider builds a fresh IcebergScanPlanProvider per call — no provider-level table reuse" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1776, + "note": "getOrLoadPropertiesResult memoized, but its one execution performs buildColumnHandles (1780→1856→getColumnHandles) AND getScanNodePropertiesResult (1801-1803) = 2 remote loads; per-split getFileFormatType is covered by this memo (claim's memoization note verified)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1202, + "note": "getSplits calls buildColumnHandles a SECOND time (un-memoized) then planScan at 1242; isBatchMode's streamingSplitEstimate at 1414; create() resolves handle (tableExists RPC) at 207" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 325, + "note": "planning flow: getFileFormatType:325 → isBatchMode:376 → getSplits:422 → per-split setFormatType(getFileFormatType()):485 — all in one createScanRangeLocations pass" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 145, + "note": "materializeLatest: resolveConnectorTableHandle:145 (tableExists RPC), beginQuerySnapshot:156, getMvccPartitionView:165 (loadTable), listLatestPartitions:181 for non-RANGE partitioned tables (another loadTable); loadSnapshot:343-346 routes the implicit query-begin pin here" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java", + "line": 988, + "note": "loadSnapshots invokes MvccTable.loadSnapshot during planning — puts materializeLatest on the plain-SELECT hot path" + } + ] + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "Every cited hop was read and verified. The heavy op is real: CatalogBackedIcebergCatalogOps.loadTable is a bare SDK catalog.loadTable() (IcebergCatalogOps.java:340-342) built raw by CatalogUtil.buildIcebergCatalog — no CachingCatalog wrap anywhere in the connector, no Table-object cache in fe-connector-iceberg / fe-connector-metastore-iceberg (property assembly only) / fe-connector-cache / fe-core (zero IcebergMetadataCache hits in fe-core main). IcebergConnector.getMetadata constructs a fresh IcebergConnectorMetadata per call (IcebergConnector.java:212-215) and the provider's resolveTable (IcebergScanPlanProvider.java:1981-1993) calls ops.loadTable unconditionally, so no layer can dedupe. Mitigations DO exist but are all partial: (a) PluginDrivenScanNode.cachedPropertiesResult memoizes the per-split getFileFormatType path down to one load-pair — it fixes the canonical per-split amplification but not the cross-site redundancy; (b) isBatchModeCache limits streamingSplitEstimate to one run — which still does its own resolveTable; (c) IcebergLatestSnapshotCache (default ON, TTL 86400s) serves ONLY beginQuerySnapshot and stores only (snapshotId, schemaId) (IcebergLatestSnapshotCache.java:57-65) — none of the other sites consult it; (d) the MVCC single-pin invariant makes materializeLatest run once per query, but that run still does 1-2 uncached loadTables; (e) IcebergScanPlanProvider.getScanNodeProperties' position-deletes branch explicitly avoids \"a second loadTable\" (1412-1417), proving the cost is known but only locally avoided. Net: on a default-config SELECT of a partitioned iceberg table, 6-7 independent remote catalog.loadTable() round trips plus 2 tableExists RPCs occur where 1 load is inherent; only beginQuerySnapshot is cached. One multiplicity correction: streaming mode REPLACES getSplits (startStreamingSplit's buildColumnHandles:1611 + streamSplits resolveTable:452 substitute for getSplits' two loads), so streaming is ~same k, not 9 as the finder claimed. Within-query redundancy is also a consistency hazard: the query pins a snapshot, but each fresh loadTable reads LATEST metadata, so a commit landing mid-planning can skew what different planning hops observe.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "loadTable = bare catalog.loadTable(toTableIdentifier(...)); catalog built raw via CatalogUtil.buildIcebergCatalog, no CachingCatalog wrap (zero grep hits in the iceberg connector)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981, + "note": "resolveTable: unconditional ops.loadTable per call, no memo field; called from getScanNodeProperties:1300, streamingSplitEstimate:410, planScan path:562, streamSplits:452" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 540, + "note": "private loadTable helper: auth-wrapped catalogOps.loadTable, no memo; getColumnHandles:587 loads fresh each call; getMvccPartitionView:1453 and listPartitions:1507 each load again; getTableHandle:324 is a remote tableExists" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1540, + "note": "beginQuerySnapshot is the ONLY cached hop: latestSnapshotCache.getOrLoad (default TTL 86400s per IcebergConnector.java:130,188-189); value is (snapshotId,schemaId) only (IcebergLatestSnapshotCache.java:57-65), never a Table" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1776, + "note": "cachedPropertiesResult memoization: per-split getFileFormatType collapsed to ONE buildColumnHandles(:1780)+getScanNodePropertiesResult pair — mitigates per-split amplification only; getSplits independently repeats buildColumnHandles:1202 + planScan:1242" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1386, + "note": "isBatchModeCache: computeBatchMode runs once, but its streamingSplitEstimate call (:1414 -> provider:410) does its own resolveTable, gated on enable_external_table_batch_mode default true" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 126, + "note": "materializeLatest (query-begin pin via loadSnapshot:343-346, single-pin invariant :114-119 = once per query): resolveConnectorTableHandle:145 (tableExists RPC) + beginQuerySnapshot:156 (cached) + getMvccPartitionView:165 (uncached loadTable) + listLatestPartitions:181/190 -> listPartitions:270 (second uncached loadTable when view non-RANGE and table has partition columns)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1600, + "note": "CORRECTION to finder: startStreamingSplit REPLACES getSplits in batch mode (buildColumnHandles:1611 + streamSplits->resolveTable provider:452 substitute for getSplits' two loads) — streaming k is ~same as sync, not 9" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 212, + "note": "getMetadata news a fresh IcebergConnectorMetadata per call — no instance-level memoization possible across the ~7 metadata calls of one planning pass" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1412, + "note": "position-deletes branch comment explicitly avoids 'a second loadTable ... one fewer remote round-trip' — cost is known in-code, avoidance is local only" + } + ], + "mitigation_found": "Partial only, none dedupes loadTable: (1) PluginDrivenScanNode.cachedPropertiesResult — fixes per-split amplification (the canonical bug) but not cross-site redundancy; (2) isBatchModeCache — one estimate run, which still loads; (3) IcebergLatestSnapshotCache (default ON, 24h TTL) — serves ONLY beginQuerySnapshot and stores (snapshotId,schemaId), not a Table; (4) MVCC single-pin invariant — materializeLatest once per query, still 1-2 uncached loads inside; (5) local second-loadTable avoidance in the position-deletes branch. NO CachingCatalog wrap, no Table-object cache at connector/metastore/fe-core/fe-connector-cache layer, and getMetadata/getScanPlanProvider build fresh objects per call.", + "corrected_multiplicity": "k-times-per-query: k≈6-7 remote catalog.loadTable() per SELECT (sync path, default config) — getMvccPartitionView(1453) + listPartitions(1507, identity-partitioned tables only) + getColumnHandles x2 (via ScanNode:1780 and :1202) + getScanNodeProperties(provider:1300) + streamingSplitEstimate(provider:410, default-on gate) + planScan(provider:562) — plus 2 tableExists RPCs (ScanNode:207, MvccTable:145). Streaming path is ~same k (startStreamingSplit REPLACES getSplits' two loads with buildColumnHandles:1611 + streamSplits:452), not 9 as the finder claimed. beginQuerySnapshot is the only cached load. 1 load is inherent => ~5-6 redundant remote loads per query per iceberg table.", + "impact_estimate": "O(k≈6-7 per query per table) x remote-IO (HMS: getTable RPC + metadata.json object-store fetch; REST: full HTTP loadTable incl. optional cred vending; Hadoop: version-hint + metadata.json reads) x ALL plugin-driven iceberg catalog flavors on EVERY SELECT under default config. At 20-500ms per loadTable this is roughly +0.1s to +3s planning latency per query and ~6-7x metastore/object-store metadata QPS amplification vs one cached load; large metadata.json (many snapshots) multiplies bytes fetched identically. Secondary: within-query metadata skew risk (each load reads LATEST while the query pins one snapshot).", + "fix_direction": "memoize (per-query) + pass-down: introduce a per-planning-pass Table memo — e.g. a (queryId, db, table)-keyed single-flight cache on the long-lived IcebergConnector (sibling of IcebergLatestSnapshotCache / the queryId-keyed IcebergRewritableDeleteStash, drained by the existing query-finish callback), consulted by IcebergConnectorMetadata.loadTable and IcebergScanPlanProvider.resolveTable. Within-query memoization is semantically safe (the query pins a snapshot anyway) and actually improves consistency. Alternative/complement: wrap the SDK catalog in CachingCatalog with TTL tied to meta.cache.iceberg.table.ttl-second and invalidation on REFRESH/commit — bigger blast radius (changes cross-query freshness semantics vs the snapshot cache), so the per-query memo is the lower-risk first step. Pass-down (threading the loaded Table from getScanNodePropertiesResult into planScan) would need SPI signature changes; the connector-internal memo avoids that." + } + }, + { + "title": "Per-query PARTITIONS metadata-table scan (O(all manifests) remote IO) on the analysis hot path, uncached, for every partitioned iceberg table", + "variant": "C-cache-bypass", + "heavy_op": "IcebergPartitionUtils.loadRawPartitions: partitionsTable.newScan().useSnapshot(id).planFiles() + task.asDataTask().rows() (IcebergPartitionUtils.java:709-718) — the PARTITIONS metadata table aggregates by reading EVERY data+delete manifest of the snapshot via FileIO", + "multiplicity": "per-query (once per statement per partitioned iceberg table, at analysis time before the scan node exists): StatementContext.loadSnapshots (StatementContext.java:988-997) memoizes only within one statement", + "entry_chain": "StatementContext.loadSnapshots:995 -> PluginDrivenMvccExternalTable.loadSnapshot:343-346 -> materializeLatest:126 -> [getMvccPartitionView PluginDrivenMvccExternalTable.java:165 -> IcebergConnectorMetadata.getMvccPartitionView:1453 loadTable -> IcebergPartitionUtils.buildMvccPartitionView:532 -> loadRawPartitions:709 (time-transform tables)] OR [listLatestPartitions PluginDrivenMvccExternalTable.java:181/190 -> :270 metadata.listPartitions -> IcebergConnectorMetadata.listPartitions:1507 loadTable -> IcebergPartitionUtils.listPartitions:631 -> loadRawPartitions:709 (all other partitioned tables)]", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 712 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 270 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1507 + } + ], + "why_heavy": "PartitionsTable.planFiles() materializes a StaticDataTask whose rows are computed by scanning ALL manifest files of the snapshot (not just the manifest list) — O(#manifests) remote avro reads plus per-entry aggregation, regardless of the query's predicate. The result is a pure function of the pinned snapshotId, and beginQuerySnapshot ALREADY serves that snapshotId from a 24h-TTL cache (IcebergConnectorMetadata.java:1536-1548, IcebergConnector.java:188-189) — yet the partition view derived from that same cached, unchanged snapshot is re-fetched remotely on every pass with no cache keyed by snapshot. Legacy master cached partition info per snapshot in the snapshot cache (IcebergUtils.loadPartitionInfo cache value). Also does 2 extra loadTable within the same pass (getMvccPartitionView + listPartitions, both already counted in finding 1).", + "gate": "only partitioned iceberg tables (unpartitioned/empty tables return early at IcebergPartitionUtils.java:605-611/632-638); runs on the implicit latest-pin path of every SELECT (explicit time-travel skips partition listing)", + "est_impact": "For a table with thousands of manifests: MBs–GBs of manifest reads and seconds of latency added to EVERY query at analysis time, even SELECT with a fully selective WHERE; scales with table history size, not with query selectivity. Likely the dominant planning cost for large partitioned tables.", + "confidence": "high", + "lens": "chain-redundancy", + "dupes": [ + "listPartitions runs a full PARTITIONS metadata-table planFiles (all manifests) on every planning pass of a partitioned iceberg table, with a deliberate no-cache [hidden-heavy-accessors]", + "MTMV freshness loops re-run a full PARTITIONS metadata-table scan + catalog loadTable PER PARTITION (no pin in refresh/mv_infos threads) [stats-partitions-freshness]" + ], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Independently re-derived the full chain by reading every hop. (1) HEAVINESS: IcebergPartitionUtils.loadRawPartitions (IcebergPartitionUtils.java:709-723) scans the PARTITIONS metadata table via partitionsTable.newScan().useSnapshot(id).planFiles() + task.asDataTask().rows() — per the SDK cost model a metadata-table scan is REMOTE IO that aggregates over EVERY data+delete manifest of the snapshot (O(#manifests) FileIO avro reads), independent of query selectivity. It is preceded by a catalogOps.loadTable (more remote IO, attributed to a separate finding). This is NOT the data-planning planFiles (that runs separately in IcebergScanPlanProvider), so it is additive, not inherent planning cost. (2) MULTIPLICITY: BindRelation.getLogicalPlan calls StatementContext.loadSnapshots for every bound relation (BindRelation.java:733); loadSnapshots memoizes only in the statement-scoped `snapshots` map (StatementContext.java:993). loadSnapshot on the implicit-latest path calls materializeLatest (PluginDrivenMvccExternalTable.java:343-346, 126), which reaches loadRawPartitions exactly once per statement via one of two branches, both verified: time-transform-eligible tables through getMvccPartitionView (materializeLatest:163-165 -> IcebergConnectorMetadata.java:1448-1454 -> buildMvccPartitionView:532, gate isValidRelatedTable:533 is in-memory specs/schema, scan at :565); all other partitioned tables through listLatestPartitions (materializeLatest:178-181/190 -> :270 metadata.listPartitions -> IcebergConnectorMetadata.java:1500-1513 -> IcebergPartitionUtils.listPartitions:631, scan at :641). So the claimed per-query multiplicity holds exactly. (3) NO MITIGATION: the 24h-TTL IcebergLatestSnapshotCache caches ONLY (snapshotId, schemaId) (IcebergConnectorMetadata.java:1540-1544, IcebergConnector.java:188-189); the IcebergManifestCache is default-off and consumed only by IcebergScanPlanProvider data planning, never by loadRawPartitions (IcebergConnector.java:159-162); no fe-connector-cache entry wraps the partition view. The result is a pure function of the pinned snapshotId, which the TTL cache deliberately keeps STABLE across queries — yet the derived view is re-fetched remotely every query: textbook variant-C cache bypass, with N = QPS over the TTL window. (4) LIVE + GATE: iceberg declares SUPPORTS_MVCC_SNAPSHOT (IcebergConnector.java:652) so tables instantiate PluginDrivenMvccExternalTable (PluginDrivenExternalDatabase.java:60-61); unpartitioned/empty tables early-return (IcebergPartitionUtils.java:605-611/632-638); explicit time-travel pins empty partition maps (PluginDrivenMvccExternalTable.java:410-411) — gate exactly as stated. (5) REGRESSION vs master confirmed: upstream-apache/master IcebergExternalMetaCache.loadSnapshotProjection caches IcebergUtils.loadPartitionInfo inside the TTL'd tableEntry value (getSnapshotCache serves it cached), so master paid the PARTITIONS scan once per TTL window for MTMV-eligible tables; for non-eligible partitioned tables the per-query scan is a new uncached cost introduced by the selectedPartitionNum feature. Both heaviness and multiplicity hold as claimed.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 712, + "note": "Heavy op verified: partitionsTable.newScan().useSnapshot(snapshotId).planFiles() then task.asDataTask().rows() (713-717) — PARTITIONS metadata-table scan = remote read of every data+delete manifest of the snapshot; javadoc at 526 itself says 'the PARTITIONS scan is a remote read'." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 565, + "note": "buildMvccPartitionView -> loadRawPartitions for time-transform (MTMV-eligible) tables; eligibility gate at 533 (isValidRelatedTable, in-memory specs/schema at 682-701) runs BEFORE the scan, so ineligible tables skip this branch and pay in listPartitions instead — exactly one scan either way." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 641, + "note": "listPartitions -> loadRawPartitions for all other partitioned tables (unpartitioned/empty early-return at 632-638; same guard for the MVCC view path at 605-611 in listPartitionNames)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1454, + "note": "getMvccPartitionView (1448-1460): catalogOps.loadTable at 1453 + buildMvccPartitionView at 1454, no cache; listPartitions (1500-1519): loadTable at 1507 + IcebergPartitionUtils.listPartitions at 1513, no cache." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1540, + "note": "beginQuerySnapshot serves the pinned snapshotId from IcebergLatestSnapshotCache; CachedSnapshot carries ONLY (snapshotId, schemaId) (1543-1544) — the snapshot id is cached/stable across queries but the partition view derived from that same id is not." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 162, + "note": "IcebergManifestCache is consumed only when meta.cache.iceberg.manifest.enable is set (default off, comment 159-160) and only by IcebergScanPlanProvider data planning (grep: no reference from IcebergPartitionUtils) — it does NOT absorb the PARTITIONS scan IO. latestSnapshotCache built at 188-189 with default 24h TTL (197-208)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 270, + "note": "materializeLatest (126) -> getMvccPartitionView (165) for RANGE tables OR listLatestPartitions (181/190) -> metadata.listPartitions (270) for other partitioned tables; explicit time-travel pins EMPTY partition maps (410-411) so it skips the scan, matching the stated gate." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java", + "line": 993, + "note": "loadSnapshots memoizes per (table, versionKey) only in the statement-scoped `snapshots` map — no cross-statement reuse; every new statement re-runs materializeLatest." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java", + "line": 733, + "note": "loadSnapshots called from getLogicalPlan for every bound table reference during analysis — the hot-path entry for every SELECT over an iceberg table." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabase.java", + "line": 61, + "note": "Path is live, not dormant: a connector with SUPPORTS_MVCC_SNAPSHOT gets PluginDrivenMvccExternalTable; iceberg declares that capability (IcebergConnector.java:652)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 704, + "note": "Legacy-comparison verified against upstream-apache/master (git show): IcebergExternalMetaCache.loadSnapshotProjection (master lines ~219-235) caches IcebergUtils.loadPartitionInfo inside the TTL'd tableEntry cache value served by getSnapshotCache (~111-115) — master paid the PARTITIONS scan once per TTL window, the SPI port pays it once per query." + } + ], + "corrected_multiplicity": "per-query as claimed: exactly one PARTITIONS metadata-table scan (plus one loadTable) per statement per distinct pinned reference of a partitioned iceberg table, at analysis time (BindRelation -> loadSnapshots), memoized only within the statement. Slightly WORSE on non-SELECT paths: getNewestUpdateVersionOrTime (dictionary poll, PluginDrivenMvccExternalTable.java:748) and isValidRelatedTable (MTMV refresh gate, :787) each call materializeLatest unconditionally, re-running the full remote enumeration per invocation with no pin reuse.", + "mitigation_found": "None on this path. IcebergLatestSnapshotCache (24h TTL) caches only (snapshotId, schemaId), not the derived partition view; IcebergManifestCache is default-off and consumed exclusively by IcebergScanPlanProvider data planning, not by loadRawPartitions' PARTITIONS scan; the fe-connector-cache framework has no partition-view entry; StatementContext memoization is statement-scoped only. The scan does run inside executeAuthenticated with the table pin threaded, but nothing dedupes it across statements despite the pinned snapshotId being deliberately stable within the TTL.", + "fix_direction": "Cache the derived partition data keyed by snapshot, mirroring master semantics: extend the per-catalog IcebergLatestSnapshotCache value (or add a sibling snapshot-id-keyed cache in the connector) to lazily carry the raw-partition list / ConnectorMvccPartitionView per (TableIdentifier, snapshotId) — the result is a pure function of snapshotId, so a HIT within the TTL serves the view with zero remote IO and stays automatically consistent with the beginQuerySnapshot pin; invalidation rides the existing TTL / REFRESH-CATALOG connector-rebuild lifecycle (IcebergConnector.java:157-162). This keeps the fix entirely plugin-side (no fe-core parsing/caching), consistent with the project's architecture rules.", + "impact_estimate": "Every SELECT (and INSERT source-read, EXPLAIN, etc.) over a partitioned iceberg table pays an O(#manifests-of-snapshot) remote manifest aggregation at analysis time, before scan planning even starts, regardless of WHERE selectivity. For a table with thousands of manifests this is MBs-GBs of remote reads and roughly seconds of added latency per query; multiplied by QPS since nothing dedupes across statements while the snapshot pin is unchanged for up to 24h. Regression vs legacy master (scan amortized to once per TTL window for MTMV-eligible tables) and an uncached new per-query cost for all other partitioned tables. Likely the dominant planning-time cost for large partitioned iceberg tables under the SPI path." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The candidate survives the mitigation hunt. The heavy op is real: IcebergPartitionUtils.loadRawPartitions (line 712) scans the PARTITIONS metadata table via partitionsTable.newScan().useSnapshot(id).planFiles() + task.asDataTask().rows(), which the SDK materializes by reading EVERY data+delete manifest of the snapshot through FileIO (the audit's ground-truth doc classifies metadata-table scans as remote IO). The entry chain is verified end-to-end: BindRelation.java:733 calls StatementContext.loadSnapshots at ANALYSIS time for every mvcc table reference -> PluginDrivenMvccExternalTable.loadSnapshot:343-346 -> materializeLatest:126, which UNCONDITIONALLY enumerates partitions for every partitioned iceberg table (RANGE-eligible time-transform tables via getMvccPartitionView:165 -> IcebergConnectorMetadata:1453 -> buildMvccPartitionView:565 -> loadRawPartitions; all other partitioned tables via listLatestPartitions:181/190 -> :270 -> IcebergConnectorMetadata.listPartitions:1507 -> IcebergPartitionUtils.listPartitions:641 -> loadRawPartitions). Exactly one PARTITIONS scan per materializeLatest either way. I hunted every cache layer and none prevents the cross-query repetition in the default config: (1) StatementContext.snapshots (993-996) memoizes per statement only; (2) IcebergLatestSnapshotCache (24h TTL default, IcebergConnector.java:130,188-189) caches only (snapshotId, schemaId) — the partition view derived from that same cached, unchanged snapshot is re-fetched remotely every query, so the cache proves the recomputed value is loop-invariant across the whole TTL window; (3) IcebergManifestCache is consumed only by the scan-provider manifest-planning path (gate meta.cache.iceberg.manifest.enable, default off) and loadRawPartitions bypasses it structurally (SDK PartitionsTable scan); (4) the iceberg SDK FileIO manifest content cache (io.manifest.cache-enabled) is derived default-OFF (IcebergCatalogFactory.java:68 DEFAULT_MANIFEST_CACHE_ENABLE=false, rule at 119-133) — when a user opts in it is only a PARTIAL mitigation (caches manifest bytes; avro decode + per-entry aggregation CPU still paid per query); (5) catalogOps.loadTable is a raw catalog.loadTable delegation (IcebergCatalogOps.java:340-342), no CachingCatalog anywhere; (6) no fe-core meta cache is interposed — materializeLatest calls the connector SPI directly. The scan path reuses the statement pin (PluginDrivenScanNode.pinMvccSnapshot:876-882 via MvccUtil.getSnapshotFromContext), so multiplicity stays per-query, not per-split. This is AMPLIFIED (not inherent) cost: the result is a pure function of the pinned snapshotId which is already served stable from a 24h cache, and the cost is paid regardless of query selectivity — a point-lookup SELECT with a fully selective WHERE still pays an O(#manifests) remote scan at analysis before planning even starts.", + "corrected_multiplicity": "per-query (>= once per statement per partitioned iceberg table, at analysis time via BindRelation.loadSnapshots; per-statement memoization prevents intra-statement repeats and the scan node reuses the pin, so it never grows to per-split — but nothing spans statements, so multiplicity = QPS x #partitioned-iceberg-tables-referenced)", + "mitigation_found": "Partial/off-by-default only. Default-on: StatementContext per-statement memoization (StatementContext.java:993-996) — bounds to once per query, does not span queries. Off-by-default: iceberg SDK FileIO manifest content cache, derived from meta.cache.iceberg.manifest.enable (or explicit io.manifest.cache-enabled) — DEFAULT_MANIFEST_CACHE_ENABLE=false (IcebergCatalogFactory.java:68, derivation 119-133); when enabled it caches manifest bytes so repeat PARTITIONS scans skip remote reads but still pay avro decode + aggregation CPU per query. Non-mitigations verified: IcebergLatestSnapshotCache caches only (snapshotId, schemaId), not the partition view; IcebergManifestCache is structurally bypassed by loadRawPartitions (SDK PartitionsTable scan) and its consumer gate is default-off; no CachingCatalog / table-metadata cache (IcebergCatalogOps.java:340-342); no fe-core cache between materializeLatest and the connector.", + "impact_estimate": "O(1 per query per partitioned iceberg table) x remote-IO O(#manifests: manifest-list + every data+delete manifest avro read, plus per-entry aggregation CPU) x broad trigger: EVERY statement that binds ANY partitioned iceberg table on the plugin-driven SPI path (both time-transform MTMV-eligible tables and identity/bucket/truncate-partitioned tables; unpartitioned/empty tables and explicit time-travel references are exempt). Paid at analysis time before planning, independent of query selectivity; scales with table history size. For a table with thousands of manifests: MBs-GBs of manifest reads and seconds of added latency per query; with the default 24h snapshot-pin TTL, every query in the window recomputes a provably identical view. Likely the dominant planning-path cost for large partitioned iceberg tables.", + "fix_direction": "memoize: cache the raw partition list / ConnectorMvccPartitionView keyed by (TableIdentifier, snapshotId) on the long-lived per-catalog IcebergConnector — either a sibling cache mirroring IcebergLatestSnapshotCache or by widening that cache's value to carry the partition view (which is exactly legacy master's IcebergSnapshotCacheValue shape: snapshot + partition info in one entry). The snapshotId key makes staleness structurally impossible; invalidate with the existing REFRESH TABLE/DATABASE/CATALOG hooks (IcebergConnector.java:524/540/552). Secondary: the two extra catalogOps.loadTable calls in getMvccPartitionView/listPartitions are subsumed by the same memoization.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 712, + "note": "Heavy op: partitionsTable.newScan().useSnapshot(snapshotId).planFiles() + asDataTask().rows() — PARTITIONS metadata-table scan reads all data+delete manifests of the snapshot via FileIO; no cache in this path" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 565, + "note": "RANGE path (time-transform tables): buildMvccPartitionView -> loadRawPartitions; eligibility gate at 533-534, unpartitioned/empty early-returns at 605-611" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 641, + "note": "LIST path (all other partitioned tables): listPartitions -> loadRawPartitions; unpartitioned/empty early-return at 632-638 (gate confirmed)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1453, + "note": "getMvccPartitionView: fresh catalogOps.loadTable + buildMvccPartitionView per call, no memoization at the connector-metadata layer" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1507, + "note": "listPartitions: fresh catalogOps.loadTable + IcebergPartitionUtils.listPartitions per call, no cache" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1540, + "note": "beginQuerySnapshot serves the snapshotId from IcebergLatestSnapshotCache — proving the partition view recomputed each query is a pure function of an already-cached, stable-for-24h input" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java", + "line": 57, + "note": "Cached value is (snapshotId, schemaId) ONLY — no partition view field; legacy master's snapshot cache value carried partition info" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 130, + "note": "DEFAULT_TABLE_CACHE_TTL_SECOND = 86400 (24h default TTL for the snapshot pin); cache constructed at 188-189" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java", + "line": 47, + "note": "Connector manifest cache consumed ONLY by IcebergScanPlanProvider manifest-level planning, gated by meta.cache.iceberg.manifest.enable default OFF; loadRawPartitions does not route through it" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE = false: SDK-level io.manifest.cache-enabled (FileIO manifest byte cache) is derived default-OFF (derivation rule at 119-133) — the only cross-query mitigation, and it is opt-in and partial (CPU still per query)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "loadTable is a raw catalog.loadTable delegation; no CachingCatalog / table-metadata cache anywhere in fe-connector-iceberg or fe-connector-metastore-iceberg (grep verified)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 165, + "note": "materializeLatest ALWAYS calls metadata.getMvccPartitionView; non-RANGE partitioned tables then call listLatestPartitions at 181/190 -> metadata.listPartitions at 270 — direct SPI calls, no fe-core cache interposed" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 343, + "note": "loadSnapshot implicit-latest pin -> materializeLatest (346): the partition enumeration runs on every plain SELECT's analysis, before the scan node exists; explicit time-travel skips listing (empty maps at 410-411)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java", + "line": 993, + "note": "snapshots map memoizes loadSnapshot per (table, versionKey) WITHIN one statement only — the sole default-on mitigation; nothing spans statements" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java", + "line": 733, + "note": "Analysis-time trigger: loadSnapshots called for every bound table reference, making the PARTITIONS scan a per-query cost for every partitioned iceberg table" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 876, + "note": "pinMvccSnapshot resolves the pin from StatementContext (882) — scan path reuses the statement pin, so the heavy op does NOT amplify to per-split; multiplicity is correctly per-query" + } + ] + } + }, + { + "title": "Same conjunct set converted to connector/iceberg expressions ~5-6 times per pass, including an unconditional EXPLAIN-only predicate serialization", + "variant": "B-chain-redundancy", + "heavy_op": "ExprToConnectorExpressionConverter.convertConjuncts (fe-core) and IcebergPredicateConverter.convert + Expression.toString serialization (connector) over the full WHERE conjunct set", + "multiplicity": "k-times-per-query (k≈5-6 with a WHERE clause): fe-core buildRemainingFilter at PluginDrivenScanNode.java:1415 (computeBatchMode), :1204 (getSplits), :1781 (getOrLoadPropertiesResult); connector-side IcebergPredicateConverter at IcebergScanPlanProvider.java:974 reached from BOTH streamingSplitEstimate:411 and planScanInternal:563 buildScan, plus :1428-1441 in getScanNodeProperties which re-converts AND stringifies every pushed predicate into the iceberg.pushdown_predicates prop for EVERY query — the prop is consumed only by appendExplainInfo (:1556-1565), i.e. wasted work for every non-EXPLAIN execution", + "entry_chain": "PluginDrivenScanNode.computeBatchMode:1415 / getSplits:1204 / getOrLoadPropertiesResult:1781 -> buildRemainingFilter:1888-1915 -> ExprToConnectorExpressionConverter.convertConjuncts; IcebergScanPlanProvider.buildScan:955-980 -> IcebergPredicateConverter:974; IcebergScanPlanProvider.getScanNodeProperties:1428-1441 -> new IcebergPredicateConverter(...).convert + StringBuilder append per predicate", + "files": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1888 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1430 + } + ], + "why_heavy": "Each conversion walks the whole conjunct expression tree, binds names against table.schema(), resolves the session zone, and (in the :1430 case) renders iceberg Expression.toString() strings — moderate CPU, not remote IO. All six results are loop-invariant for the pass (same conjuncts, same schema, same zone) but nothing memoizes them: buildRemainingFilter recomputes into no cache, and the two SPI methods share no provider instance state by design (fresh provider per call, IcebergConnector.java:583-592).", + "gate": "queries with a WHERE clause; the :1430 EXPLAIN-prop serialization additionally requires filter.isPresent() and a non-system table", + "est_impact": "Low — microseconds to low milliseconds per query for typical predicates, growing with conjunct count and IN-list size; pure CPU on the planning thread. Reported because it compounds with findings 1/4 (each redundant conversion sits behind its own redundant loadTable).", + "confidence": "high", + "lens": "chain-redundancy", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Independently re-traced every hop. (1) Multiplicity holds or is WORSE than claimed: per query with a WHERE clause the same conjunct set is converted 6-7 times (up to 8 with the iceberg manifest cache enabled). fe-core side (Expr->ConnectorExpression via stateless ExprToConnectorExpressionConverter.convertConjuncts): convertPredicate->buildFilterConstraint (doFinalize, once), computeBatchMode->buildRemainingFilter (once, isBatchModeCache), getSplits/startSplit/startStreamingSplit->buildRemainingFilter (once, mutually exclusive paths), getOrLoadPropertiesResult->buildRemainingFilter (once, then cached). buildRemainingFilter (:1888-1915) has no memoization — each call rewalks all conjuncts. Connector side (ConnectorExpression->iceberg Expression via stateless IcebergPredicateConverter, fresh provider per SPI call per IcebergConnector:583-592): streamingSplitEstimate:411->buildScan:974 (runs per query — enable_external_table_batch_mode defaults true), planScanInternal:563->buildScan:974, getScanNodeProperties:1428-1441 (convert + Expression.toString), plus a claim-missed 4th site: planFileScanTaskWithManifestCache:1715->combineFilter:1795 re-converts inside the SAME planScan that already converted in buildScan (manifest-cache gate). (2) The :1428-1441 serialization into PUSHDOWN_PREDICATES_PROP is confirmed EXPLAIN-only waste: grep shows its sole consumer is appendExplainInfo:1556-1565, and getScanNodeProperties runs for EVERY query via the cached getOrLoadPropertiesResult on the toThrift/per-split path. (3) Heaviness is exactly as claimed — moderate CPU only (recursive tree walk, in-memory schema field binds, zone resolve, string rendering; no remote IO in the conversion itself; the co-located resolveTable/loadTable costs belong to separate findings). (4) The per-split loop does NOT amplify this: getFileFormatType:527-534 reads the cached props map, so multiplicity stays k-per-query as claimed. Both heaviness (as claimed) and multiplicity (as claimed or worse) hold, and the impact honestly stays Low (microseconds-to-low-ms CPU on the planning thread, growing with conjunct count/IN-list size).", + "corrected_multiplicity": "k-times-per-query with k ≈ 6-7 typical (4 fe-core Expr->ConnectorExpression conversions + 2-3 connector ConnectorExpression->iceberg Expression conversions), up to 8 when the iceberg manifest cache is enabled (combineFilter at IcebergScanPlanProvider.java:1795 re-converts within the same planScan that already converted in buildScan:974 — a site the original claim missed). Never per-split: the per-split getFileFormatType path is memoized via cachedPropertiesResult.", + "mitigation_found": "Partial memoization exists but only collapses repeats WITHIN two of the sites, not across sites: isBatchModeCache (PluginDrivenScanNode.java:1385-1390) makes computeBatchMode run once, and cachedPropertiesResult (:1776-1810) makes the getOrLoadPropertiesResult->getScanNodeProperties chain run once (so the per-split getFileFormatType loop at FileQueryScanNode.java:485 costs only a map lookup, :527-534). Nothing memoizes the conversion RESULT itself: buildRemainingFilter recomputes on every call, both converter classes are stateless, and the connector provider is rebuilt fresh per SPI call (IcebergConnector.java:583-592), so no instance-level caching is possible across streamingSplitEstimate/planScan/getScanNodePropertiesResult.", + "fix_direction": "Two independent, low-risk hoists: (a) fe-core — cache buildRemainingFilter's Optional in a field alongside filteredToOriginalIndex, invalidated at the same point convertPredicate already invalidates cachedPropertiesResult/scanNodeProperties (PluginDrivenScanNode.java:796-798), collapsing 4 fe-core conversions to at most 2 (pre/post applyFilter); (b) connector — make getScanNodeProperties' PUSHDOWN_PREDICATES_PROP serialization lazy or explain-gated (e.g. only when the session is an EXPLAIN, or reuse the predicates already converted by buildScan in the same pass), and let planFileScanTaskWithManifestCache reuse buildScan's converted predicate list instead of re-converting via combineFilter. Per the project's no-parsing-in-fe-core rule, (b) stays entirely in the connector.", + "impact_estimate": "Low — pure planning-thread CPU, microseconds to low milliseconds per query for typical predicates; scales with conjunct count and IN-list size (each of the ~6-8 passes rewalks every literal). No remote IO is added by the redundancy itself. Worth fixing opportunistically because it compounds with the co-located redundant resolveTable/loadTable calls (separate findings) and the EXPLAIN-only serialization is 100% wasted on every non-EXPLAIN execution.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1914, + "note": "buildRemainingFilter (:1888-1915) calls ExprToConnectorExpressionConverter.convertConjuncts with no caching; recomputed on every call" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1204, + "note": "getSplits call site of buildRemainingFilter (sync path); :1528 startSplit and :1613 startStreamingSplit are the mutually exclusive batch-path equivalents" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1415, + "note": "computeBatchMode passes buildRemainingFilter() inline to streamingSplitEstimate; memoized to once per query via isBatchModeCache :1385-1390" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1781, + "note": "getOrLoadPropertiesResult converts once more, then caches the SPI result in cachedPropertiesResult (:1777-1809)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 774, + "note": "convertPredicate -> buildFilterConstraint -> convertConjuncts (:1879); claim omitted this 4th fe-core conversion site; invoked from FileQueryScanNode.doFinalize:252 every query" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 974, + "note": "buildScan constructs a new IcebergPredicateConverter and converts the filter on every call; reached from streamingSplitEstimate:411, streamSplits:453, planScanInternal:563, planSystemTableScan:749" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1430, + "note": "getScanNodeProperties re-converts (:1428-1441) and stringifies every pushed predicate into PUSHDOWN_PREDICATES_PROP for every query with a filter on a non-system table" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1556, + "note": "grep confirms appendExplainInfo (:1556-1565) is the ONLY consumer of PUSHDOWN_PREDICATES_PROP — the :1430 work is wasted on non-EXPLAIN executions" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1795, + "note": "claim-missed extra site: planFileScanTaskWithManifestCache:1715 -> combineFilter re-converts the same filter inside the same planScan that already converted in buildScan (manifest-cache gate)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 583, + "note": "getScanPlanProvider (:583-592) builds a fresh provider per call — no shared instance state to memoize conversions across SPI calls, as the claim stated" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPredicateConverter.java", + "line": 128, + "note": "convert() is a stateless recursive tree walk with per-conjunct bind-check against the in-memory schema — moderate CPU, no remote IO, no caching" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 527, + "note": "getFileFormatType reads the cached props map (getOrLoadScanNodeProperties), so the FileQueryScanNode.java:485 per-split call does NOT amplify the conversions — multiplicity correctly stays k-per-query" + } + ] + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "No mechanism dedupes the conversions in the default configuration. The two memoizations that exist (isBatchModeCache at PluginDrivenScanNode.java:1386, cachedPropertiesResult at :1777) only ensure each ENTRY POINT runs once; the conversion itself is recomputed at every entry point: buildRemainingFilter (:1888) rebuilds ExprToConnectorExpressionConverter.convertConjuncts (:1914) fresh on each of its 3-per-query call sites (:1204/:1415/:1781; batch paths swap :1204 for :1528/:1613), and convertPredicate adds a 4th fe-core conversion via buildFilterConstraint (:774 -> :1879) that the finder missed. Connector-side there is no provider-instance state to memoize into: IcebergConnector.getScanPlanProvider (IcebergConnector.java:583-592) builds a fresh IcebergScanPlanProvider per call, PluginDrivenScanNode.resolveScanProvider (:224-226) does not cache it, and grep confirms zero cache/memo fields in ExprToConnectorExpressionConverter (366 lines, static pure) and IcebergPredicateConverter (900 lines, fresh instance per use). buildScan converts at IcebergScanPlanProvider.java:974 from both streamingSplitEstimate (:411, fires whenever computeBatchMode runs with slots since enable_external_table_batch_mode defaults true at :407) and planScanInternal (:563) or streamSplits (:453). getScanNodeProperties performs a 7th conversion PLUS Expression.toString serialization (:1428-1441) into PUSHDOWN_PREDICATES_PROP whose sole consumer is appendExplainInfo (:1556-1565, rendered only at VERBOSE EXPLAIN) — verified the :1428 gate is only !systemTable && filter.isPresent(), so this is pure wasted work for every non-EXPLAIN execution. fe-connector-cache is a Caffeine metadata-cache framework and holds nothing expression-related; the manifest-cache combineFilter conversion (:1715 -> :1794) is an ADDITIONAL one but gated off by default (DEFAULT_MANIFEST_CACHE_ENABLE=false, IcebergCatalogFactory.java:68). Caveat for any fix: conjuncts mutate mid-pass (convertPredicate clears them at :786; pruneConjunctsFromNodeProperties rewrites them at :1767-1768) and buildRemainingFilter has the filteredToOriginalIndex side effect, so a memo must reuse the existing invalidation points at :796-798. Severity stays low (local CPU, micro-to-low-ms), which is why this is CONFIRMED-but-minor rather than a headline finding.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1914, + "note": "buildRemainingFilter recomputes ExprToConnectorExpressionConverter.convertConjuncts on every call; no cached field for the result (method body :1888-1915)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1204, + "note": "buildRemainingFilter call #1 in getSplits (batch paths substitute :1528 startSplit / :1613 startStreamingSplit — still one per query)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1415, + "note": "call #2 in computeBatchMode; isBatchModeCache (:1386-1389) caps it at once per query — partial mitigation, does not dedupe across entry points" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1781, + "note": "call #3 in getOrLoadPropertiesResult; cachedPropertiesResult (:1777) caps it at once — partial mitigation only" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 774, + "note": "MISSED BY FINDER: convertPredicate -> buildFilterConstraint (:1878-1881) is a 4th full-conjunct convertConjuncts per query, raising k to ~7" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 974, + "note": "buildScan converts filter via new IcebergPredicateConverter each call; reached per query from BOTH streamingSplitEstimate:411 (enable_external_table_batch_mode default true at :407) and planScanInternal:563 (or streamSplits:453 on the streaming path)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1430, + "note": "getScanNodeProperties re-converts AND Expression.toString-serializes every pushed predicate into PUSHDOWN_PREDICATES_PROP (:1428-1441); gate is only !systemTable && filter.isPresent() — runs for every execution, not just EXPLAIN" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1556, + "note": "sole consumer of PUSHDOWN_PREDICATES_PROP is appendExplainInfo (:1556-1565), rendered only at VERBOSE EXPLAIN per the :1544 comment — confirms wasted work on the non-EXPLAIN hot path" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 583, + "note": "getScanPlanProvider builds a FRESH IcebergScanPlanProvider per call (:583-592), so no provider-instance memo can survive; PluginDrivenScanNode.resolveScanProvider (:224-226) does not cache it either" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE=false — the extra combineFilter conversion (IcebergScanPlanProvider.java:1715 -> :1794) only fires when the manifest cache is explicitly enabled" + } + ], + "corrected_multiplicity": "k≈7 per query with a WHERE clause in default config (not 5-6): 4x fe-core convertConjuncts (convertPredicate:774, computeBatchMode:1415, getSplits:1204 or batch equivalent, getOrLoadPropertiesResult:1781) + 2x connector IcebergPredicateConverter via buildScan:974 (streamingSplitEstimate:411 + planScanInternal:563) + 1x getScanNodeProperties:1430 convert-and-stringify (EXPLAIN-only consumer). +1 (combineFilter:1794) when the manifest cache is enabled (off by default).", + "mitigation_found": "Partial only: isBatchModeCache (PluginDrivenScanNode.java:1386) and cachedPropertiesResult (:1777) cap each entry point at one invocation but nothing dedupes the conversion across the 7 sites; no cache/memo exists in ExprToConnectorExpressionConverter or IcebergPredicateConverter (grep-verified); provider instances are rebuilt per resolveScanProvider() call (IcebergConnector.java:583-592) so no connector-side state survives; fe-connector-cache covers metadata caches only. No gate skips the :1430 EXPLAIN-prop serialization for non-EXPLAIN executions.", + "impact_estimate": "O(k≈7 per query) x local-CPU (expression-tree walk + schema.findField binding + session-zone resolve + toString rendering; scales with conjunct count and IN-list size) x broad trigger (every iceberg query with a WHERE clause; the :1430 serialization additionally every non-system-table query with a pushed filter). Absolute cost microseconds to low milliseconds per query on the planning thread — Low severity standalone; matters mainly because conversions 5-7 each sit inside a method that also does its own resolveTable/loadTable (separate findings).", + "fix_direction": "Memoize + pass-down: cache buildRemainingFilter's Optional in a node field invalidated exactly where cachedPropertiesResult already is (convertPredicate :796-798 and after pruneConjunctsFromNodeProperties mutates conjuncts :1767-1768, since buildRemainingFilter also side-effects filteredToOriginalIndex); connector-side, extract a shared convert-once helper so buildScan and getScanNodeProperties reuse one List within a call, and gate the PUSHDOWN_PREDICATES_PROP stringification on an explain signal (or derive it from the same converted list) instead of reconverting unconditionally." + } + }, + { + "title": "getTableComment does a full remote loadTable per table inside the information_schema.tables / SHOW TABLE STATUS per-table loop", + "variant": "A-loop-amplification", + "heavy_op": "IcebergCatalogOps.loadTable() (HMS getTable RPC + metadata.json object-store GET, or REST loadTable roundtrip) just to read the 'comment' table property", + "multiplicity": "per-table: FrontendServiceImpl.listTableStatus loops `for (TableIf table : tables)` (FrontendServiceImpl.java:719) and unconditionally calls table.getComment() at :755 (NOT gated by needTableStatusColumn) — N tables => N serial remote table loads per one information_schema.tables / SHOW TABLE STATUS request, executed under the table read lock", + "entry_chain": "FrontendServiceImpl.listTableStatus loop fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:719 -> status.setComment(table.getComment()) :755 -> PluginDrivenExternalTable.getComment fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java:944 -> metadata.getTableComment :952 -> IcebergConnectorMetadata.getTableComment fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java:305 -> loadTable(new IcebergTableHandle(db,tbl)) :313 -> IcebergCatalogOps.loadTable fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java:340-341 (raw catalog.loadTable, no CachingCatalog wrap anywhere in IcebergCatalogFactory)", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 313 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java", + "line": 952 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java", + "line": 755 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 341 + } + ], + "why_heavy": "Each call is a full catalog table load: HMS getTable thrift RPC plus reading the current metadata JSON from the object store (or one REST loadTable roundtrip). Legacy IcebergExternalTable.getComment read the property off the already-cached table object; here every getComment() is a fresh remote load with no cache at any layer (no CachingCatalog, no connector-side table cache, no comment memoization on the fe-core table object).", + "est_impact": "A db with N iceberg tables pays N serial remote loads (typically 50-300ms each) for a single `SELECT * FROM information_schema.tables` or SHOW TABLE STATUS — hundreds of tables => tens of seconds to minutes, on a path BI tools hit constantly; also hammers the metastore/REST service.", + "gate": "none — comment column is set unconditionally on this path", + "confidence": "high", + "lens": "hidden-heavy-accessors", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Every hop re-derived from source. (1) Heaviness holds: IcebergConnectorMetadata.getTableComment loads the full iceberg Table via the private loadTable seam, which is a bare catalog.loadTable (IcebergCatalogOps.java:340-341) — remote IO per the SDK cost model (HMS getTable thrift RPC + current metadata.json object-store GET, or one REST loadTable roundtrip) — only to read table.properties().get(\"comment\") which would be in-memory on an already-loaded table. (2) Multiplicity holds: FrontendServiceImpl.listTableStatus iterates all tables of the db (:720) and calls table.getComment() unconditionally at :755, BEFORE the first needTableStatusColumn gate (:756), serially, under table.readLock() (:749) — so N iceberg tables = N serial remote loads per information_schema.tables / SHOW TABLE STATUS request, even when the projection doesn't include TABLE_COMMENT. (3) No mitigation at any layer: PluginDrivenExternalTable.getComment (:944-961) has no memoization and no isObjectCreated() gate (its sibling getCachedRowCount at :927-936 explicitly has one to keep this path non-blocking, proving the pattern was known); IcebergConnector.getMetadata (:212-215) builds a fresh IcebergConnectorMetadata per op; grep for CachingCatalog across fe-connector-iceberg is empty; the only table-keyed cache (IcebergLatestSnapshotCache) stores only (snapshotId, schemaId) and is consulted solely by beginQuerySnapshot (IcebergConnectorMetadata.java:1540), never by getTableComment; the manifest cache is FileIO manifest-file-level and does not cover the metadata.json read. The comment in getTableComment itself (:306-307) admits it wraps a 'remote load'. Cost is unconditional and on a path BI tools poll constantly. Trivial correction: the loop is at FrontendServiceImpl.java:720, not :719.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java", + "line": 720, + "note": "per-table loop `for (TableIf table : tables)` in listTableStatus (candidate said :719; actual :720)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java", + "line": 755, + "note": "status.setComment(table.getComment()) — unconditional, placed BEFORE the first needTableStatusColumn gate at :756, inside table.readLock() taken at :749" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java", + "line": 952, + "note": "getComment(boolean) calls metadata.getTableComment(session, remoteDbName, tableName) fresh every invocation; no memoized field, no isObjectCreated() gate (unlike getCachedRowCount at :927-936 which gates exactly to keep SHOW TABLE STATUS non-blocking)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 313, + "note": "getTableComment -> loadTable(new IcebergTableHandle(dbName, tableName)); javadoc at :306-307 says 'Wrap the remote load in the auth context'; returns table.properties().getOrDefault(\"comment\",\"\") at :314" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 540, + "note": "private loadTable = context.executeAuthenticated(() -> catalogOps.loadTable(db, tbl)) — no cache consulted" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 341, + "note": "impl is bare catalog.loadTable(toTableIdentifier(...)) — remote IO; grep for CachingCatalog across fe-connector-iceberg returns zero hits" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 212, + "note": "getMetadata builds a NEW IcebergConnectorMetadata per operation, so even instance-level memoization would not survive across the per-table calls" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java", + "line": 55, + "note": "the only table-keyed connector cache stores only (snapshotId, schemaId) pins; consumed solely by beginQuerySnapshot (IcebergConnectorMetadata.java:1540), never by getTableComment — no mitigation" + } + ], + "corrected_multiplicity": "per-table (N iceberg tables in the db => N serial remote catalog.loadTable calls per single information_schema.tables / SHOW TABLE STATUS request, each under that table's read lock; repeated on every such request — no TTL cache absorbs it)", + "mitigation_found": "None on this path. IcebergLatestSnapshotCache caches only (snapshotId, schemaId) for beginQuerySnapshot; manifest cache (default-off) is FileIO manifest-level and irrelevant to loadTable's metadata.json read; no CachingCatalog wrap; fe-core getComment has neither memoization nor the isObjectCreated() non-blocking gate its sibling getCachedRowCount has. Only softener: exceptions degrade to \\\"\\\" (LOG.debug), so failures don't error the query — but the remote call is still made.", + "fix_direction": "Two complementary layers: (a) fe-core: gate comment retrieval on needTableStatusColumn(requiredColumns, \\\"COMMENT\\\") in FrontendServiceImpl.listTableStatus (and/or add an isObjectCreated()-style non-blocking guard in PluginDrivenExternalTable.getComment, mirroring getCachedRowCount); (b) connector: serve getTableComment through a per-catalog MetaCacheEntry (same fe-connector-cache framework as IcebergLatestSnapshotCache, keyed by TableIdentifier with the meta.cache.iceberg.table TTL), or extend the latest-snapshot pin's cached value to carry the comment property so the pinned load is reused.", + "impact_estimate": "A db with N iceberg tables pays N serial remote loads (HMS getTable RPC + metadata.json GET, typically ~50-300ms each) per information_schema.tables / SHOW TABLE STATUS request — 200 tables ≈ 10-60s wall time, on a path BI tools and SHOW TABLE STATUS poll constantly; also multiplies metastore/REST/object-store load, and each load holds the table's read lock while blocking on remote IO." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "Every layer of the cited chain was read and no mitigation exists in the default configuration. FrontendServiceImpl.listTableStatus loops per table (:720) and sets comment unconditionally (:755) — the comment column is NOT gated by needTableStatusColumn, unlike ENGINE/CREATE_TIME/etc., and runs under table.readLock() (:749), so the remote IO is serial and holds the read lock. PluginDrivenExternalTable.getComment (:944-961) has no cached field and delegates every call to metadata.getTableComment. IcebergConnectorMetadata.getTableComment (:313) does a fresh loadTable per call via the raw seam (:540-547) into IcebergCatalogOps.loadTable (:340-341), which is a bare catalog.loadTable — grep confirms zero CachingCatalog usage in fe-connector-iceberg and fe-connector-metastore-iceberg, and the metastore-iceberg providers contain no Caffeine/getOrLoad/CacheBuilder. The two existing caches do not apply: IcebergLatestSnapshotCache only stores (snapshotId, schemaId) pins for the snapshot-pin path (used at IcebergConnectorMetadata:1540, not by getTableComment); the io.manifest.cache-enabled derivation in IcebergCatalogFactory feeds iceberg's FileIO manifest-content cache, which does not cover the HMS getTable RPC or the metadata.json GET that loadTable performs. The SPI default (ConnectorTableOps:336-339) returns \"\", so this remote load is an iceberg-override opt-in with no compensating cache. Each getComment = 1 HMS thrift getTable + 1 object-store metadata.json GET (or 1 REST loadTable roundtrip). N iceberg tables in a db => N serial remote loads per listTableStatus RPC; a full information_schema.tables scan issues one RPC per database, so O(total tables in catalog) remote loads per query.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java", + "line": 755, + "note": "status.setComment(table.getComment()) unconditional inside the per-table loop (for at :720), under table.readLock() taken at :749; contrast ENGINE at :756+ which IS gated by needTableStatusColumn" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java", + "line": 952, + "note": "getComment(boolean) delegates every call to metadata.getTableComment; no cached comment field on the table object, exceptions swallowed to \"\"" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 313, + "note": "getTableComment does loadTable(new IcebergTableHandle(db,tbl)) per call just to read table.properties().get(\"comment\"); private loadTable at :540-547 is a raw authenticated seam call with no memoization" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 341, + "note": "CatalogBackedOps.loadTable = bare catalog.loadTable(toTableIdentifier(...)); no CachingCatalog wrap anywhere in fe-connector-iceberg or fe-connector-metastore-iceberg (grep zero hits)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java", + "line": 93, + "note": "Only caches CachedSnapshot (snapshotId+schemaId) pins via getOrLoad; getTableComment does not route through it, so this cache is not a mitigation for the comment path" + }, + { + "file": "fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java", + "line": 336, + "note": "SPI default getTableComment returns \"\" — the remote load is an iceberg-specific override opt-in with no compensating cache layer" + } + ], + "mitigation_found": "None on this path. Checked and ruled out: no CachingCatalog wrap in IcebergCatalogFactory or metastore-iceberg providers; IcebergLatestSnapshotCache caches only snapshot/schema-id pins for the snapshot-pin path and is not consulted by getTableComment; IcebergManifestCache / io.manifest.cache-enabled caches manifest FILE content in FileIO and does not cover the HMS getTable RPC or metadata.json GET inside catalog.loadTable; no comment field memoized on PluginDrivenExternalTable; fe-core ExtMetaCacheMgr is used for row counts on the same class but not for comments; no needTableStatusColumn gate on the COMMENT column in FrontendServiceImpl.", + "corrected_multiplicity": "per-table: N serial remote loadTable calls per listTableStatus RPC (one RPC covers one database); a full information_schema.tables scan issues one RPC per database, so O(total iceberg tables in the catalog) remote loads per query. Minor line correction: the loop `for (TableIf table : tables)` is at FrontendServiceImpl.java:720 (finding cited 719); all other cited lines verified exact.", + "impact_estimate": "O(tables-per-db) x remote-IO (1 HMS getTable thrift RPC + 1 object-store metadata.json GET, or 1 REST loadTable roundtrip, typically 50-300ms each) x ALL iceberg plugin-catalog tables with no gate, triggered by every SHOW TABLE STATUS and every information_schema.tables access — a path BI tools (Tableau/PowerBI/DBeaver schema sync) hit constantly. 200 tables => roughly 10-60s serial latency per request plus metastore/REST hammering; the remote IO additionally happens while holding each table's read lock, delaying any concurrent writer needing that lock. Legacy IcebergExternalTable.getComment read the property off the already-cached table object, so this is a regression versus master, not inherent cost.", + "fix_direction": "pass-down/memoize: serve the comment from a table load that already happens and is cached — either (a) capture the comment into the fe-core table object / schema-cache value at table build time (getTableSchema already loads the same Table at IcebergConnectorMetadata:348) and have getComment read the stored value, or (b) add a connector-side table-metadata cache (extend IcebergLatestSnapshotCache's pattern or wrap the SDK catalog in CachingCatalog keyed by meta.cache.iceberg.table.ttl-second) so getTableComment hits the cache. Secondary hardening: gate status.setComment with needTableStatusColumn(\"COMMENT\") in FrontendServiceImpl like the neighboring columns, so projections that skip the comment column pay nothing." + } + }, + { + "title": "One SELECT planning pass re-loads the same iceberg table remotely 4-6 times through getter/resolver-named accessors (getColumnHandles, resolveTable, getScanNodeProperties, listPartitions)", + "variant": "B-chain-redundancy", + "heavy_op": "IcebergCatalogOps.loadTable() (raw catalog.loadTable — HMS getTable RPC + metadata.json GET, or REST loadTable roundtrip; no CachingCatalog wrap, verified in IcebergCatalogFactory)", + "multiplicity": "k-times-per-query, k≈4-6 in one planning pass: (1) getColumnHandles called twice — PluginDrivenScanNode.buildColumnHandles at getSplits:1202 (or startSplit:1526 / startStreamingSplit:1611) AND getOrLoadPropertiesResult:1780, buildColumnHandles (:1856-1859) has no memoization and each call loads the table at IcebergConnectorMetadata.java:587; (2) IcebergScanPlanProvider.resolveTable re-loads at streamingSplitEstimate:410 + streamSplits:452 (batch path) or planScan:562, and again at getScanNodeProperties:1300; (3) listPartitions loads once more (:1507). Only beginQuerySnapshot is cached (IcebergLatestSnapshotCache, IcebergConnectorMetadata.java:1540). Write planning repeats the same pattern: IcebergWritePlanProvider resolveTable at :241, :257, :286 per statement", + "entry_chain": "FileQueryScanNode.doFinalize fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java:248-253 -> PluginDrivenScanNode.getSplits buildColumnHandles fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java:1202 -> IcebergConnectorMetadata.getColumnHandles fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java:587 loadTable [load #1]; planScan -> IcebergScanPlanProvider.resolveTable fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java:562 -> :1981-1989 ops.loadTable [load #2]; PluginDrivenScanNode.getOrLoadPropertiesResult :1780 buildColumnHandles [load #3] -> :1801 getScanNodePropertiesResult -> IcebergScanPlanProvider.getScanNodeProperties :1300 resolveTable [load #4]; batch path adds streamingSplitEstimate :410 + streamSplits :452 [loads #5-6]; each -> IcebergCatalogOps.loadTable fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java:341", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 587 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1202 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1780 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1300 + } + ], + "why_heavy": "Every loadTable is a full remote metadata resolution (metastore RPC + current metadata JSON fetch; a full REST roundtrip for REST catalogs). The table state is loop-invariant across one planning pass (the query is even pinned to one snapshot by beginQuerySnapshot), yet no layer memoizes the loaded Table: not the SDK (no CachingCatalog), not IcebergCatalogOps (thin passthrough, IcebergCatalogOps.java:340-341), not the providers, not the scan node (only the properties MAP is cached, and buildColumnHandles is rebuilt fresh each call). For Kerberos catalogs each resolveTable also wraps a NEW FileIO (wrapTableForScan :2018-2029), guaranteeing SDK manifest-cache misses across the repeated loads.", + "est_impact": "Adds 3-5 redundant remote metadata roundtrips (~50-300ms each) to EVERY iceberg query's planning — roughly 0.2-1.5s extra planning latency per query at any QPS, plus proportional metastore/REST service load; worst on REST catalogs where a comment in the code says table reload 'per query' is the freshness design, but the actual rate is 4-6x per query.", + "gate": "none for the 4x baseline (all SELECTs); batch/streaming path and Kerberos add the extra loads/cache misses", + "confidence": "high", + "lens": "hidden-heavy-accessors", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Independently re-derived the full chain; both heaviness and multiplicity hold, and the default-path multiplicity is actually WORSE than claimed. Heaviness: every load bottoms out in IcebergCatalogOps.loadTable (IcebergCatalogOps.java:340-341), a raw catalog.loadTable — remote IO per the SDK cost model (HMS getTable RPC + metadata.json GET, or full REST roundtrip). No CachingCatalog anywhere in fe-connector-iceberg or fe-connector-metastore-iceberg (grep confirmed; factory builds raw catalogs via CatalogUtil.buildIcebergCatalog / direct initialize), and no Table memoization in IcebergConnectorMetadata.loadTable (:540-543), IcebergScanPlanProvider.resolveTable (:1981-1993), or the scan node. Multiplicity for ONE non-batch SELECT (default session vars): (1) FileQueryScanNode.createScanRangeLocations:325 calls getFileFormatType BEFORE anything else -> PluginDrivenScanNode:527 -> getOrLoadPropertiesResult:1780 buildColumnHandles (:1856-1859, rebuilt fresh every call) -> IcebergConnectorMetadata.getColumnHandles:587 loadTable [load 1]; (2) same pass :1801-1803 getScanNodePropertiesResult -> SPI default (ConnectorScanPlanProvider.java:455-461) -> IcebergScanPlanProvider.getScanNodeProperties:1300 resolveTable [load 2]; (3) FileQueryScanNode:376 isBatchMode -> computeBatchMode:1412-1415 calls streamingSplitEstimate on EVERY query with output slots (gate at IcebergScanPlanProvider:407 is only sys-table or enable_external_table_batch_mode=false, default TRUE) -> :410 resolveTable [load 3] — the candidate wrongly counted this as batch-only, so the baseline is 5 not 4; it additionally reads snapshot.dataManifests(table.io()) (:424-425), extra manifest-list IO on every query; (4) FileQueryScanNode:422 getSplits -> PluginDrivenScanNode:1202 buildColumnHandles -> getColumnHandles:587 [load 4]; (5) :1242-1244 planScan -> planScanInternal:562 resolveTable [load 5]. Partitioned tables add a 6th: Nereids pruning -> ExternalTable.initSelectedPartitions:468-475 -> PluginDrivenExternalTable.getNameToPartitionItems:807-831 (comment explicitly says \"no FE-side partition-value cache... lists partitions per query\") -> IcebergConnectorMetadata.listPartitions:1507 loadTable PLUS a remote PARTITIONS metadata-table scan. Batch path substitutes startStreamingSplit:1611 buildColumnHandles [load] + streamSplits:452 resolveTable [load], same count. The table state is loop-invariant within the pass (query is snapshot-pinned via beginQuerySnapshot). Kerberos amplifier confirmed: wrapTableForScan (:2018-2029) wraps a NEW IcebergAuthenticatedFileIO per resolveTable, so iceberg's ManifestFiles content cache (keyed by FileIO instance) misses across the repeated loads. Write path confirmed too: IcebergWritePlanProvider resolveTable at :241/:257/:286 -> own raw ops.loadTable (:689-697). What is NOT broken (mitigations that keep this k-per-query, not per-split): cachedPropertiesResult memoizes the properties result (PluginDrivenScanNode:1776-1810) so the per-split getFileFormatType loop (FileQueryScanNode:485) does NOT re-load — the canonical #64134 amplification is fixed; isBatchModeCache (:1386) prevents repeat estimates; beginQuerySnapshot is served from IcebergLatestSnapshotCache (IcebergConnectorMetadata:1540), but that caches only the snapshot/schema id pin, never the Table.", + "corrected_multiplicity": "5 remote loadTable calls per SELECT planning pass on an unpartitioned table with default session vars (claim said 4: streamingSplitEstimate at computeBatchMode:1412-1415 fires on EVERY slotted query, not just the batch path); 6 for partitioned tables (listPartitions via Nereids pruning, plus its extra PARTITIONS metadata-table scan). Batch/streaming path is also ~6. Not per-split — the per-split getFileFormatType loop is properly memoized via cachedPropertiesResult.", + "mitigation_found": "Partial mitigations exist but none memoize the loaded Table: cachedPropertiesResult (PluginDrivenScanNode.java:1776-1810) caches the scan-node-properties RESULT so the per-split loop is not amplified; isBatchModeCache (:1385-1389) runs streamingSplitEstimate once; IcebergLatestSnapshotCache (IcebergConnectorMetadata.java:1540) caches only the snapshot-id/schema-id pin. No CachingCatalog, no per-query Table cache at any layer.", + "fix_direction": "Hoist/memoize the resolved Table for the planning pass: (a) in PluginDrivenScanNode, compute buildColumnHandles once and reuse across getSplits/getOrLoadPropertiesResult/startSplit (drops 1 load generically for all connectors); (b) in IcebergScanPlanProvider/IcebergConnectorMetadata, cache the loaded Table keyed by (queryId, db.table, pinned snapshot) — the query is already snapshot-pinned by beginQuerySnapshot, so a per-query Table memo is semantics-preserving — or wrap the raw catalog in the SDK CachingCatalog with the existing meta.cache.iceberg.table.ttl-second knob; (c) under Kerberos, reuse one wrapped FileIO per resolved Table so the manifest ContentCache stops missing.", + "impact_estimate": "4-5 redundant remote metadata resolutions (metastore RPC + metadata.json GET, or REST roundtrips, ~50-300ms each) added to EVERY iceberg SELECT's planning, i.e. roughly 0.2-1.5s extra planning latency per query plus 5-6x the intended metastore/REST load; partitioned tables additionally pay a redundant PARTITIONS metadata-table scan per query. Write statements pay the same pattern 2-3x.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 341, + "note": "loadTable = raw catalog.loadTable(toTableIdentifier(...)) — thin passthrough, remote IO per SDK cost model; no CachingCatalog anywhere in the connector (grep of fe-connector-iceberg + fe-connector-metastore-iceberg: zero hits)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 587, + "note": "getColumnHandles loads the table fresh on every call (via loadTable:540-543, auth wrapper only, no cache)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1856, + "note": "buildColumnHandles has no memoization — calls metadata.getColumnHandles every time; invoked at getSplits:1202, getOrLoadPropertiesResult:1780, startSplit:1526, startStreamingSplit:1611" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1780, + "note": "getOrLoadPropertiesResult (first call) does buildColumnHandles [load] then getScanNodePropertiesResult:1801-1803 [load]; only the RESULT is cached (cachedPropertiesResult:1777), so per-split getFileFormatType (:527-528) is NOT amplified — this finding is k-per-query, not per-split" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 325, + "note": "createScanRangeLocations calls getFileFormatType() up front (triggers loads 1-2), isBatchMode() at :376 (load 3), getSplits at :422 (loads 4-5); per-split setFormatType at :485 hits the memoized properties" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1414, + "note": "computeBatchMode calls streamingSplitEstimate for EVERY query with slots — corrects the claim's 'batch path adds' framing; gate is only enable_external_table_batch_mode (default true, IcebergScanPlanProvider:407)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981, + "note": "resolveTable -> ops.loadTable per call, no memo; called from streamingSplitEstimate:410, streamSplits:452, planScanInternal:562, getScanNodeProperties:1300; Kerberos wrapTableForScan:2018-2029 builds a NEW FileIO wrapper per call (manifest ContentCache keyed by FileIO instance -> guaranteed miss)" + }, + { + "file": "fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java", + "line": 455, + "note": "default getScanNodePropertiesResult wraps getScanNodeProperties — iceberg does not override it, so the resolveTable at IcebergScanPlanProvider:1300 fires through the scan node's cached-result path exactly once per query" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java", + "line": 831, + "note": "getNameToPartitionItems -> metadata.listPartitions per query, comment states 'no FE-side partition-value cache (per CACHE-P1)'; reached from ExternalTable.initSelectedPartitions:468-475 (Nereids pruning) for partitioned tables -> IcebergConnectorMetadata:1507 loadTable + remote PARTITIONS metadata-table scan" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1540, + "note": "beginQuerySnapshot is the ONLY cached load (IcebergLatestSnapshotCache, TTL-gated) — and it caches only the snapshot/schema id pin, never the Table object, so it cannot serve the other 5-6 loads" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java", + "line": 689, + "note": "write path has its own raw resolveTable (-> ops.loadTable:693/697), called at :241, :257, :286 — same redundancy pattern per write statement" + } + ] + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "Hunted every layer for an existing mitigation and found only partial ones; none prevents the repeated remote loadTable in the default configuration. (1) No SDK-level cache: IcebergCatalogFactory never wraps CachingCatalog (grep negative across the file), and IcebergCatalogOps.CatalogBackedIcebergCatalogOps.loadTable is a raw catalog.loadTable passthrough (IcebergCatalogOps.java:340-342). (2) No connector-level memoization: IcebergConnector.getMetadata builds a NEW IcebergConnectorMetadata per call (IcebergConnector.java:212-214), so instance state could not help; IcebergConnectorMetadata.loadTable (540-547) and IcebergScanPlanProvider.resolveTable (1981-1993) load fresh every call. (3) The only load-avoiding cache on the path is IcebergLatestSnapshotCache (default TTL 24h, IcebergConnector.java:188-209) and it caches ONLY the (snapshotId, schemaId) pin for beginQuerySnapshot (IcebergConnectorMetadata.java:1540-1545) — never the Table. (4) PluginDrivenScanNode.cachedPropertiesResult (1776-1810) is a real memo but only caps the properties result at one computation: it prevents the canonical per-split amplification (FileQueryScanNode.java:485 getFileFormatType per split resolves from the cached map after the first split) — the finding is correctly k-per-query, not per-split. (5) IcebergManifestCache is default OFF (IcebergConnector.java:157-160 comment: consumed only when meta.cache.iceberg.manifest.enable is set). (6) fe-core has a schema cache, but buildColumnHandles (PluginDrivenScanNode.java:1856-1873, no memo) bypasses it and calls connector getColumnHandles which loads the table (IcebergConnectorMetadata.java:587). (7) The partition path is per-query by explicit design with no FE cache (PluginDrivenExternalTable.java:826-831, CACHE-P1 comment) and loads the table again (IcebergConnectorMetadata.java:1507). I additionally found one load the finder MISSED: computeBatchMode calls streamingSplitEstimate for every slotted query (PluginDrivenScanNode.java:1412-1420) and the provider's gate defaults ON (sessionBool(ENABLE_EXTERNAL_TABLE_BATCH_MODE, true), IcebergScanPlanProvider.java:407) before resolveTable at :410 — so even queries that end up on the sync path pay that load plus a manifest-list metadata read. Default sync-path count for an unpartitioned SELECT is therefore 5 loadTable roundtrips (batch probe + buildColumnHandles in getSplits:1202 + planScan resolveTable:562 + buildColumnHandles again:1780 + getScanNodeProperties resolveTable:1300), 6 for partitioned (+listPartitions, which also runs a PARTITIONS metadata-table scan), of which 1 is inherent. Kerberos claim verified: wrapTableForScan (IcebergScanPlanProvider.java:2018-2029) builds a NEW IcebergAuthenticatedFileIO/BaseTable per resolveTable, so any per-FileIO SDK content cache would miss across the repeated loads. The table state is loop-invariant within the pass (the query is pinned to one snapshot by beginQuerySnapshot), so 4-5 of the loads are pure redundancy.", + "corrected_multiplicity": "k-times-per-query with k=5 for an unpartitioned SELECT in the DEFAULT config (batch-mode probe fires on every slotted query since enable_external_table_batch_mode defaults true — one load the candidate attributed only to the batch path), k=6 for partitioned tables (+listPartitions load + PARTITIONS metadata scan via Nereids pruning), similar k≈5-6 on the true batch/streaming path (probe + startStreamingSplit buildColumnHandles:1611 + streamSplits:452 + the two propsResult loads). Roughly 4-5 of the k are redundant (1 load is inherent to planning). beginQuerySnapshot's load is the only cached one (24h-TTL pin cache, default on). Per-split amplification does NOT occur (cachedPropertiesResult memo). Write path adds ~3 more per INSERT (IcebergWritePlanProvider resolveTable at :241/:257/:286 — not re-verified line-by-line this pass).", + "mitigation_found": "Four partial mechanisms, none sufficient: (1) IcebergLatestSnapshotCache (default 24h TTL) — caches only the (snapshotId, schemaId) pin for beginQuerySnapshot, not the Table; saves 1 load/query. (2) PluginDrivenScanNode.cachedPropertiesResult + scanNodeProperties memo (1776-1823) — caps getScanNodeProperties+its buildColumnHandles at once per node, preventing per-split amplification only. (3) isBatchModeCache + cached streamingSplitEstimate field (157-166, 1385-1390) — the batch probe runs once, not twice (dispatch+explain). (4) IcebergManifestCache / io.manifest.cache-enabled — DEFAULT OFF (only when user sets meta.cache.iceberg.manifest.*), and mitigates manifest reads, not loadTable; under Kerberos the per-resolveTable FileIO rewrap defeats per-FileIO SDK caches anyway. No CachingCatalog wrap, no session/query-scoped table memo, no fe-core cache on the buildColumnHandles or listPartitions paths (CACHE-P1 explicitly forgoes one).", + "impact_estimate": "O(k-per-query, k=5-6) x remote-IO (each load = HMS getTable RPC + metadata.json GET, or a full REST loadTable roundtrip; partitioned tables add a PARTITIONS metadata-table scan) x ALL iceberg SELECTs on every catalog flavor — no gate. Net redundancy ~4-5 metadata roundtrips per query (~50-300ms each depending on catalog/latency), i.e. roughly 0.2-1.5s avoidable planning latency per query plus 5-6x the intended metastore/REST request load. Worst for REST catalogs (full HTTP roundtrip per load) and high-QPS point queries where planning dominates.", + "fix_direction": "Memoize the loaded Table at query/planning-pass scope: the snapshot pin from beginQuerySnapshot already makes the load loop-invariant and provides a natural cache key (TableIdentifier + pinned snapshotId). Cheapest surgical form: (a) memoize buildColumnHandles' result in PluginDrivenScanNode (called 2-3x with identical output per node), and (b) a query-scoped (or short-TTL, pin-keyed) Table memo inside the connector shared by resolveTable/loadTable/listPartitions — e.g. on IcebergConnector next to latestSnapshotCache, invalidated with it. Alternatively hoist: resolve the Table once in the provider entry and pass it down; under Kerberos also reuse one wrapTableForScan wrapper per query so SDK FileIO caches can hit.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 341, + "note": "loadTable = raw catalog.loadTable(toTableIdentifier(...)) passthrough; no CachingCatalog anywhere in IcebergCatalogFactory (grep negative)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 212, + "note": "getMetadata(session) constructs a NEW IcebergConnectorMetadata per call — no instance-level memo possible; lines 157-162: manifest cache 'consumed only when meta.cache.iceberg.manifest.enable is set (default off)'; 188-209: snapshot-pin cache default TTL 24h" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 587, + "note": "getColumnHandles loads the table (loadTable/loadSysTable) on every call; private loadTable at 540-547 has no memoization" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1856, + "note": "buildColumnHandles: fresh connector.getMetadata + getColumnHandles each call, no memo; called at 1202 (getSplits), 1780 (getOrLoadPropertiesResult), 1526/1611 (batch paths)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1414, + "note": "computeBatchMode calls scanProvider.streamingSplitEstimate on EVERY slotted query (isBatchModeCache only dedups repeat isBatchMode() calls) — an extra loadTable the candidate attributed only to the batch path" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 407, + "note": "streamingSplitEstimate gate: sessionBool(ENABLE_EXTERNAL_TABLE_BATCH_MODE, true) — default ON — then resolveTable at :410; streamSplits resolveTable :452; planScan resolveTable :562; getScanNodeProperties resolveTable :1300" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981, + "note": "resolveTable: catalogOpsResolver.apply(session) then ops.loadTable every call, no memo; wrapTableForScan at 2018-2029 builds a NEW IcebergAuthenticatedFileIO/BaseTable per call (Kerberos) defeating per-FileIO SDK caches" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1776, + "note": "PARTIAL MITIGATION verified: cachedPropertiesResult memoizes getScanNodePropertiesResult once per node, so FileQueryScanNode.java:485's per-split getFileFormatType does NOT amplify — redundancy is k-per-query, not per-split" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1540, + "note": "beginQuerySnapshot is the ONLY cached load: IcebergLatestSnapshotCache.getOrLoad caches just (snapshotId, schemaId), never the Table object" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java", + "line": 826, + "note": "getNameToPartitionItems: 'no FE-side partition-value cache (per CACHE-P1)' — Nereids pruning of partitioned tables calls metadata.listPartitions per query, which loads the table again (IcebergConnectorMetadata.java:1507) plus a PARTITIONS metadata-table scan" + } + ] + } + }, + { + "title": "getScanNodeProperties infers file_format_type via whole-table planFiles() fallback — DORIS-27138 pattern resurrected at per-query multiplicity", + "variant": "B-chain-redundancy", + "heavy_op": "IcebergWriterHelper.inferFileFormatFromDataFiles() -> table.newScan().planFiles() (manifest-list + manifest remote IO) just to read the FIRST data file's format", + "multiplicity": "k-times-per-query (one EXTRA whole-table manifest scan per planning pass, on top of planScan's own planFiles at IcebergScanPlanProvider.java:627/1657/1660 which already yields dataFile.format() per file); repeats on EVERY query/EXPLAIN of the table", + "entry_chain": "FileQueryScanNode.createScanRangeLocations (fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java:325) -> PluginDrivenScanNode.getFileFormatType (fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java:527) -> getOrLoadScanNodeProperties (PluginDrivenScanNode.java:1815) -> getOrLoadPropertiesResult (PluginDrivenScanNode.java:1801) -> IcebergScanPlanProvider.getScanNodeProperties (fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java:1311) -> IcebergWriterHelper.getFileFormat (fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java:265) -> resolveFileFormatName (IcebergWriterHelper.java:284) -> inferFileFormatFromDataFiles (IcebergWriterHelper.java:287) -> table.newScan().planFiles() (IcebergWriterHelper.java:291)", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1311 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java", + "line": 291 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1801 + } + ], + "why_heavy": "planFiles() reads the snapshot's manifest-list plus manifests through FileIO (remote object-store IO, ParallelIterable prefetches manifests beyond the first task). The node-level memoization (verified: cachedPropertiesResult, PluginDrivenScanNode.java:1776-1823) caps it at once per scan node, but it is pure redundancy: planScanInternal's own planFiles enumeration already surfaces every file's real format, and each range re-emits its own per-file format anyway (IcebergScanRange.populateRangeParams:421-425); the scan-level value only picks BE's V1/V2 scanner. This is the exact getFileFormat->planFiles fallback the problem-class doc calls out (PR #64134), ported into the connector and now sitting on EVERY query's mandatory getScanNodeProperties path.", + "gate": "Only when the table carries neither the 'write-format' nor the 'write.format.default' property (IcebergWriterHelper.java:278-284) — common for migrated tables and engine-created tables that never set the property; non-system tables only", + "est_impact": "Doubles the per-query metadata IO for gated tables: +1 full manifest scan per SELECT/EXPLAIN (hundreds of ms to seconds on large tables or slow object stores), multiplied by QPS", + "confidence": "high", + "lens": "per-split-serialization", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Every hop of the claimed chain re-derived by reading the code. (1) Heaviness holds: IcebergWriterHelper.inferFileFormatFromDataFiles does table.newScan().planFiles() (IcebergWriterHelper.java:291) — remote IO (manifest-list + manifests via FileIO) per the SDK cost model. Nuance: it reads only the FIRST FileScanTask, so the guaranteed floor is manifest-list + one manifest fetch rather than a full manifest scan; however the connector's own docs (IcebergScanPlanProvider.java:1999-2003) confirm multi-manifest tables fan manifest reads onto iceberg's ParallelIterable worker pool, which prefetches beyond the first task, so cost approaches a whole-table manifest scan on large tables. (2) Multiplicity holds exactly as claimed: PluginDrivenScanNode's cachedPropertiesResult memo (PluginDrivenScanNode.java:1776-1810) verifiably caps the op at once per scan-node instance (the per-split call at FileQueryScanNode.java:485 is defused), but nothing caches across queries or per (table,snapshot) — every SELECT/EXPLAIN of a gated table pays one extra planFiles. (3) Redundancy holds: the same planning pass already runs its own planFiles (IcebergScanPlanProvider.java:627 -> splitFiles at 1657/1660) whose FileScanTasks each expose file().format(), re-emitted per range at IcebergScanRange.java:421-425; the scan-level file_format_type (BE V1/V2 scanner selector, comment at IcebergScanPlanProvider.java:1303-1309) is derivable from data the pass already fetched. (4) Live path, non-system tables only, gated on the table lacking both 'write-format' and 'write.format.default' explicit properties (IcebergWriterHelper.java:278-284) — the exact migrated-table population PR #64134 targeted; empty tables (currentSnapshot()==null) short-circuit with no IO (line 288-289). No mitigating cache at any other layer: IcebergManifestCache only intercepts planFileScanTask when meta.cache.iceberg.manifest.enable is on (default off, comment at 1675-1679), and IcebergWriterHelper's raw newScan().planFiles() bypasses it regardless; resolveTable's loadTable may be metastore-cached but the SDK never caches planFiles results. This is variant B (chain redundancy) plus the doc's 'disguised light accessor' cause: getFileFormat reads like a property lookup and hides a manifest scan.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java", + "line": 291, + "note": "inferFileFormatFromDataFiles: try (CloseableIterable files = table.newScan().planFiles()) — remote manifest IO; reads first task's format only; lines 288-289 short-circuit snapshot-less tables to parquet with no IO" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java", + "line": 278, + "note": "Gate: fallback reached only when explicit table properties contain neither 'write-format' (line 278) nor TableProperties.DEFAULT_FILE_FORMAT ('write.format.default', line 281) — resolveFileFormatName:277-284" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1311, + "note": "getScanNodeProperties puts file_format_type = IcebergWriterHelper.getFileFormat(table) for every non-system table; comment at 1303-1309 explains BE selects FileScannerV2 vs V1 from this scan-level value" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1776, + "note": "getOrLoadPropertiesResult: cachedPropertiesResult null-check memoization (1777-1809) verified — heavy op runs once per scan-node instance, NOT per split; but cache dies with the node, so each query/EXPLAIN re-pays" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 527, + "note": "getFileFormatType override reads getOrLoadScanNodeProperties().get(PROP_FILE_FORMAT_TYPE) — the memoized path the per-split caller hits" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 325, + "note": "createScanRangeLocations calls getFileFormatType() at scan level; line 485 calls it again per split inside splitToScanRange (rangeDesc.setFormatType(getFileFormatType())) — both defused by the node memo" + }, + { + "file": "fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java", + "line": 455, + "note": "SPI default getScanNodePropertiesResult wraps getScanNodeProperties (460-461); grep shows iceberg does not override it (only fe-connector-es does), so the chain from PluginDrivenScanNode:1802 lands in IcebergScanPlanProvider.getScanNodeProperties" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1657, + "note": "splitFiles: scan.planFiles() at 1657 (forced file_split_size) and 1660 (heuristic path) — the planning pass's own inherent manifest scan, whose FileScanTasks already carry each file's real format; reached from planScanInternal via planFileScanTask at line 627" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 421, + "note": "populateRangeParams re-emits each range's own per-file format (orc/parquet, lines 421-425), overriding the scan-level default — proving per-file format info survives to BE independently of the scan-level probe" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1999, + "note": "Class comment: planFiles reads manifest list + manifests 'on the CALLING thread for small tables and fanned onto iceberg's shared worker pool (ParallelIterable) for multi-manifest tables' — confirms prefetch amplification beyond the first task in inferFileFormatFromDataFiles" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1676, + "note": "planFileScanTask doc: manifest cache only engages when meta.cache.iceberg.manifest.enable is on (default off); IcebergWriterHelper's raw table.newScan().planFiles() bypasses it in any case — no mitigating cache layer" + } + ], + "corrected_multiplicity": "k-times-per-query as claimed, where k = number of PluginDrivenScanNode instances over the table in the plan (1 per table occurrence; self-join = 2). Node-level memo caps at once per node; no cross-query or per-(table,snapshot) cache, so it recurs on every SELECT/EXPLAIN of a gated table. Heavy-op cost per occurrence: floor = manifest-list + 1 manifest remote read (only first FileScanTask consumed); worst case approaches full manifest scan via ParallelIterable prefetch on multi-manifest tables.", + "mitigation_found": "Partial only: PluginDrivenScanNode.cachedPropertiesResult (PluginDrivenScanNode.java:1776-1810) reduces the DORIS-27138 per-split amplification to once per scan node — already acknowledged by the candidate. No further layer helps: no per-(table,snapshot) memo of the inferred format in the connector, IcebergManifestCache is default-off and bypassed by the raw newScan().planFiles() anyway, and the SDK never caches planFiles results even when the Table object itself comes from a metastore cache. Empty tables (no current snapshot) are exempt (IcebergWriterHelper.java:288-289).", + "fix_direction": "Remove the redundant probe by reusing information the pass already has or memoizing per snapshot: (a) derive the scan-level format from planScan's own enumeration (e.g. first FileScanTask of the SplitPlan) instead of a fresh newScan().planFiles() — requires plumbing since getScanNodeProperties currently runs independently of planScan; or (b) memoize the inferred format keyed by (table identity, currentSnapshot().snapshotId()) in the connector (fe-connector-cache framework fits), since a snapshot's first data file format is immutable; or (c) at minimum, bound the probe (newScan().option to limit manifest reads / use snapshot.summary() hints) — do NOT delete the fallback itself, it exists deliberately for migrated ORC tables (PR #64134).", + "impact_estimate": "+1 remote manifest read sequence (manifest-list + >=1 manifest, up to ~full manifest scan via ParallelIterable prefetch) per query/EXPLAIN per scan node, on every non-system iceberg table lacking explicit write-format/write.format.default properties — i.e. migrated tables and engine-created tables that never persisted the property (table.properties() holds only explicit entries, so this population is common). On object stores this is roughly +2 to +N HTTP GETs and tens of ms to seconds of added planning latency per query, multiplied by QPS; up to ~2x the query's total metadata IO in the worst case. Not a correctness issue; cost scales with manifest count and store latency." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The heavy op and chain are real and no default-config mitigation prevents the repeated cost. IcebergScanPlanProvider.getScanNodeProperties (line 1311) calls IcebergWriterHelper.getFileFormat for EVERY non-system iceberg table on the mandatory scan-node-properties path; when the table has neither 'write-format' nor 'write.format.default' (IcebergWriterHelper.java:278-284), it falls to inferFileFormatFromDataFiles -> table.newScan().planFiles() (line 291) = remote manifest-list + manifest IO (ParallelIterable may prefetch many manifests even though only the first task is consumed). Mitigation hunt results: (a) PluginDrivenScanNode's node-level memo IS effective — cachedPropertiesResult (1776-1809) computes once per scan node; the convertPredicate invalidation (line 797) fires before the first consumer (doFinalize order: convertPredicate at FileQueryScanNode.java:252, first consumption at :325), so the per-split loop (FileQueryScanNode.java:485) never re-triggers it; this caps the finding at k-per-query, exactly as the candidate claims, but does not eliminate the extra planFiles. (b) Doris IcebergManifestCache is disabled by default (DEFAULT_MANIFEST_CACHE_ENABLE=false, IcebergScanPlanProvider.java:201) and, even when enabled, is wired only into planFileScanTaskWithManifestCache (1683-1693) — the helper's raw planFiles bypasses it entirely. (c) The Iceberg SDK manifest content cache (io.manifest.cache-enabled) is derived to true only from the same off-by-default FE spec or an explicit user key (IcebergCatalogFactory.java:119-133). (d) No (table,snapshot)-keyed memo of the resolved format exists at any layer; IcebergCatalogOps.loadTable (340-341) is a raw delegation. Redundancy confirmed: the same planning pass's split enumeration already runs scan.planFiles (627 -> 1657/1660) and each FileScanTask carries file().format(), and IcebergScanRange re-emits per-file format per range — so the extra planFiles duplicates information the pass already fetches. This is a faithful port of the legacy IcebergUtils.getFileFormat (PR #64134 fallback) with the per-split amplification fixed but the per-query redundant whole-manifest-path scan retained. The same un-memoized fallback also re-executes on each write-planning call site (IcebergWritePlanProvider.java:400/464/525/746, IcebergConnectorTransaction.java:658/690), giving multiple planFiles per INSERT on gated tables — lower multiplicity, same pattern. One correction to the candidate: the gate is broader than migrated tables — any table whose creating engine never explicitly set a write-format property (common; engines don't persist the default) triggers the fallback.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java", + "line": 291, + "note": "inferFileFormatFromDataFiles: table.newScan().planFiles() remote manifest IO just to read first data file's format; gate = missing write-format/write.format.default at lines 278-284" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1311, + "note": "getScanNodeProperties puts file_format_type via IcebergWriterHelper.getFileFormat(table) for every non-system table — mandatory per-query path" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1801, + "note": "cachedPropertiesResult memo verified: single getScanNodePropertiesResult SPI call per scan node per planning pass (caps multiplicity at k-per-query, does not remove the extra planFiles)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 797, + "note": "convertPredicate invalidates the memo, but runs BEFORE the first consumer (doFinalize: convertPredicate then createScanRangeLocations), so still exactly one computation" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 485, + "note": "per-split rangeDesc.setFormatType(getFileFormatType()) hits the node memo — per-split amplification of DORIS-27138 is NOT resurrected; first heavy call is the hoisted one at line 325" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 201, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE = false — Doris manifest cache off by default" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1683, + "note": "manifest cache, when enabled, is wired only into planFileScanTaskWithManifestCache — the helper's raw newScan().planFiles() never consults it" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 130, + "note": "SDK io.manifest.cache-enabled derived to true only when off-by-default meta.cache.iceberg.manifest.enable spec is on (or user sets the SDK key explicitly)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1660, + "note": "splitFiles' scan.planFiles() (also 1657; invoked from planFileScanTask at 627) — the same planning pass already enumerates every FileScanTask with file().format(), making the helper's planFiles pure redundancy" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 341, + "note": "loadTable is a raw catalog.loadTable delegation — no cached-format or table-level memo layer above the helper" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java", + "line": 400, + "note": "write paths (also 464/525/746 and IcebergConnectorTransaction 658/690) re-run the same un-memoized getFileFormat fallback per call — same pattern at write multiplicity" + } + ], + "corrected_multiplicity": "k-times-per-query where k = number of iceberg scan nodes over gated tables in the plan (typically 1): one EXTRA planFiles (manifest-list + >=1 manifest, with ParallelIterable prefetch) per scan node per planning pass, on top of the split enumeration's inherent planFiles; NOT per-split (node memo verified). Additionally 1-4x per write-planning pass on the sink/transaction call sites.", + "mitigation_found": "Partial only: (1) PluginDrivenScanNode.cachedPropertiesResult memoizes per scan node — caps at once per planning pass but does not remove the redundant planFiles; (2) Doris IcebergManifestCache — off by default and not wired into the helper's raw planFiles even when on; (3) Iceberg SDK io.manifest.cache-enabled content cache — off by default (derived from the same disabled FE spec). No (table,snapshot)-keyed format memo at any layer; no pass-down from the split enumeration that already knows every file's format.", + "impact_estimate": "O(k-per-query, k~1) x remote-IO (1 manifest-list read + 1..N manifest reads per occurrence; hundreds of ms to seconds on large tables / slow object stores) x trigger breadth: any iceberg table lacking BOTH 'write-format' and 'write.format.default' properties (migrated tables plus tables whose creating engine never persisted a write-format default) — for those tables it roughly doubles per-SELECT/EXPLAIN metadata IO, multiplied by QPS; zero impact on tables carrying the property.", + "fix_direction": "Memoize the resolved format in the connector keyed by (table UUID, current snapshot id) — snapshot-invariant and safe across queries; or pass-down: derive the scan-level file_format_type from the split enumeration's FileScanTasks (each already carries file().format()); note the ordering constraint that getScanNodeProperties is consumed before getSplits (FileQueryScanNode.createScanRangeLocations:325), so pass-down requires either reordering or a lazily-resolved property, making the (table,snapshot) memo the lower-risk fix. Cheapest correct-enough alternative: cap the fallback to reading only the first manifest via table.currentSnapshot().dataManifests(io).get(0) + ManifestFiles.read instead of full planFiles." + } + }, + { + "title": "buildRange recomputes per-file/per-partition invariants (partition JSON, identity map, delete-file conversion) for every byte-slice split task", + "variant": "D-loop-invariant-not-hoisted", + "heavy_op": "IcebergPartitionUtils.getPartitionDataJson (serializePartitionValue date/time formatting + Jackson writeValueAsString) + getIdentityPartitionInfoMap + LinkedHashMap reordering + buildDeleteFiles (per-delete bounds decode + carrier build) per FileScanTask", + "multiplicity": "per-split: TableScanUtil.splitFiles slices one data file into k tasks (IcebergScanPlanProvider.java:1657/1670) and the loop at :627-637 (and streaming :516-519) calls buildRange per task; results are identical for all k slices of one file, and partitionDataJson/identityMap are identical for ALL files sharing one (specId, partition tuple)", + "entry_chain": "IcebergScanPlanProvider.planScanInternal (fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java:627) -> buildRangeForTask (:689) -> buildRange (:1045): getPartitionDataJson (:1056) -> IcebergPartitionUtils.getPartitionDataJson (fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java:208-216) + getIdentityPartitionInfoMap (:1060 -> IcebergPartitionUtils.java:142-175) + ordered-map rebuild (:1061-1067) + buildDeleteFiles (:1117 -> :1129-1174)", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1056 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 208 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1117 + } + ], + "why_heavy": "Each call does per-field transform dispatch, LocalDate/LocalDateTime formatting with zone conversion for timestamp partitions (IcebergPartitionUtils.java:363-388), a Jackson JSON serialization, two schema findColumnName lookups per field, plus a fresh carrier object per delete file. Individually µs-scale, but it is pure loop-invariant recomputation: no memo keyed on (specId, PartitionData) or on the parent DataFile exists at any layer, so a 100k-split scan performs ~100k Jackson serializations where the distinct-partition count (often 10-1000) would do. Per-split CPU cost (not payload — the bytes per range are required by the wire format).", + "gate": "Partitioned tables for the JSON/identity-map part (partitioned && dataFile.partition() instanceof PartitionData, :1051); delete conversion applies to any v2+ table with merge-on-read deletes", + "est_impact": "~5-15µs x N_splits: ~0.5-2s extra planning CPU + significant garbage at 100k splits on a timestamp-partitioned MOR table; negligible under 1k splits", + "confidence": "high", + "lens": "per-split-serialization", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Independently re-derived the full chain. (1) Multiplicity holds: both the eager loop (IcebergScanPlanProvider.planScanInternal :628-637) and the streaming source (:516-518) call buildRangeForTask -> buildRange once per FileScanTask, and tasks are byte-offset slices from TableScanUtil.splitFiles (:1657 forced-size path, :1670 heuristic path) — k slices per data file, each SplitScanTask delegating file()/deletes() to the same parent, so partition tuple and delete list are bit-identical across all k slices. (2) Recomputation holds: buildRange (:1045-1122) recomputes per task the partition_data_json (Jackson writeValueAsString over a per-field serializePartitionValue pass, IcebergPartitionUtils :183-216), the identity-partition map (second per-field pass with findColumnName per identity field, :142-175), a LinkedHashMap reorder (:1061-1067), and fresh delete carriers with ByteBuffer bounds decoding (buildDeleteFiles :1129-1139, readPositionBound :1183-1193). Timestamp/date partitions additionally do LocalDate/LocalDateTime formatting with zone conversion per call (:363-388). (3) No hoist/memoize anywhere: the only computeIfAbsent in the provider (:1814) is a ManifestEvaluator cache on the unrelated manifest-cache planning path; nothing is keyed on (specId, PartitionData) or on the parent DataFile. (4) Heaviness qualifies under the class doc's variant D (line 59 explicitly lists repeated in-loop parsing/string-building/identical-substructure construction), with the honest caveat — stated by the candidate itself — that each call is µs-scale CPU, not remote IO; aggregate impact is real only at large split counts. Two minor overstatements found, neither verdict-changing: \"two findColumnName lookups per field\" is actually one per identity field inside the loop (getIdentityPartitionColumns is correctly hoisted to :455/:576, and findColumnName is an in-memory lazy-indexed lookup); and legacy IcebergScanNode.createIcebergSplit did the same per-split work, so this is legacy parity, not a migration regression (not a refutation under the class rules, but relevant for prioritization). Gate is as claimed and not rare: partitioned && partition() instanceof PartitionData (:1051) is the normal partitioned-table case; delete conversion fires on any v2 merge-on-read table.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 628, + "note": "Eager per-task loop: for (FileScanTask task : plan.tasks) -> buildRangeForTask(:631); tasks are byte-slices from splitFiles" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 517, + "note": "Streaming path IcebergStreamingSplitSource.hasNext() calls buildRangeForTask per task — same recomputation, no memo" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1657, + "note": "TableScanUtil.splitFiles(scan.planFiles(), fileSplitSize) — k byte-offset slices per data file (forced-size path); heuristic path at :1670" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1056, + "note": "buildRange recomputes getPartitionDataJson per task; gate at :1051 (partitioned && instanceof PartitionData) is the normal partitioned-table case" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1060, + "note": "getIdentityPartitionInfoMap recomputed per task, then LinkedHashMap reorder :1061-1067 — all loop-invariant per (specId, partition tuple)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1117, + "note": "buildDeleteFiles per task -> convertDelete per delete (:1135-1136) with readPositionBound ByteBuffer decode (:1183-1193); identical for all k slices of one file (SplitScanTask.deletes() delegates to parent)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 208, + "note": "getPartitionDataJson = getPartitionValues (per-field serializePartitionValue, :183-201) + Jackson mapper().writeValueAsString (:211) — fresh serialization every call" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 142, + "note": "getIdentityPartitionInfoMap: second per-field pass, one findColumnName per identity field (:161) + serializePartitionValue (:168) — candidate's 'two findColumnName per field' slightly overstated" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 385, + "note": "TIMESTAMP partitions: per-call zone conversion + ISO formatting (DATE :368, TIME :375) — heaviest per-field case" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1814, + "note": "Only memo in the provider (evalCache.computeIfAbsent) is for ManifestEvaluator on the manifest-cache path — confirms NO cache exists for partition JSON/identity map/delete carriers" + }, + { + "file": "plan-doc/perf-heavy-op-hot-path-problem-class.md", + "line": 59, + "note": "Variant D explicitly covers in-loop repeated parsing/string-building/identical-substructure construction — the µs-scale-per-call op qualifies under the class's own definition" + } + ], + "corrected_multiplicity": "per-split (per FileScanTask after byte-slicing). Partition JSON + identity map are invariant per (specId, partition tuple) — N_splits recomputations where distinct-partition count (typically 10-1000) would suffice; delete-carrier conversion is invariant per parent DataFile — N_splits recomputations where N_files would suffice (and the same global delete files repeat across files). Applies identically on both eager and streaming paths.", + "mitigation_found": "None on this path. The provider's only memo (evalCache at IcebergScanPlanProvider.java:1806-1814) serves ManifestEvaluator on the manifest-cache planning path, unrelated to buildRange. getIdentityPartitionColumns IS correctly hoisted (:455/:576), showing the pattern exists but was not applied to the per-task invariants. Note: legacy IcebergScanNode.createIcebergSplit had the same per-split behavior, so this is inherited parity, not a migration regression.", + "fix_direction": "Thread a per-scan memo through buildRangeForTask (single shared choke point for both eager and streaming paths): (a) an IdentityHashMap keyed on task.file() (all k slices share the parent DataFile instance) caching {partitionSpecId, partitionDataJson, orderedPartitionValues, deleteCarrierList} — collapses per-split to per-file with trivially correct keying; (b) optionally a second map keyed on (specId, StructLikeWrapper.wrap(partitionData)) to further dedup partition JSON/identity map across files sharing a partition tuple — needs StructLikeWrapper for correct StructLike equality. The streaming source would hold the memo as an instance field; the eager path a local passed into the loop. Bound memory by the distinct-file/partition count already materialized in planning.", + "impact_estimate": "Order-of-magnitude agrees with the candidate: roughly 3-10µs CPU + allocation garbage per split on a timestamp-partitioned table (two per-field serialization passes, zone conversion, one small Jackson serialization, map churn, per-delete carrier rebuild). ~0.3-1s extra planning CPU plus GC pressure at 100k splits; sub-10ms and irrelevant below ~1k splits. Pure planning-thread CPU, no remote IO — a real but second-tier finding versus IO-class amplifications." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The recompute is real and unmitigated at the claimed tier in the default configuration. Both enumeration paths (eager loop IcebergScanPlanProvider.java:628-633 and streaming source :516-518) call buildRangeForTask -> buildRange once per SLICED FileScanTask, and TableScanUtil.splitFiles (:1657 forced, :1670 heuristic; default target 64MB per :136, 32MB initial per :135) slices one data file into k tasks that share the same DataFile/PartitionData/deletes — so partition JSON (buildRange:1056), identity map (:1060), the LinkedHashMap reorder (:1061-1067), and the delete-file conversion (:1117 -> :1129-1174) are recomputed k times per file with byte-identical results, and the JSON/identity-map pair is additionally identical across ALL files sharing one (specId, partition tuple). Mitigation hunt results: (a) scan-level invariants ARE hoisted (formatVersion :575, orderedPartitionKeys :576, zone :577, partitioned :578, vendedToken :586) — the per-scan tier is handled, but no memo exists at the (specId, partition) or per-DataFile tier: the only scan-path callers of getPartitionDataJson/getIdentityPartitionInfoMap are buildRange itself, and grep finds no cache/computeIfAbsent/IdentityHashMap around them in the provider, IcebergPartitionUtils, fe-connector-cache, or fe-connector-metastore-iceberg; (b) IcebergManifestCache is off by default (DEFAULT_MANIFEST_CACHE_ENABLE=false, :201) and orthogonal — its path feeds the same buildRange unchanged (:1702-1703); (c) SDK-internal caches blunt per-call cost but not repetition: Schema.findColumnName (IcebergPartitionUtils.java:161) is a map lookup against iceberg's per-Schema cached idToName index, JsonUtil.mapper() (:211) is a static singleton, task.deletes() is a cached list with no IO (:1090-1091 comment) — yet the Jackson writeValueAsString, timestamp zone-conversion formatting (IcebergPartitionUtils.java:376-388), and per-delete ByteBuffer decode + carrier allocation (:1156-1174, :1183-1193) still run per slice; (d) fe-core FileQueryScanNode.java:475-478 only consumes the connector-provided map. Severity caveat (honest framing): the heavy op is µs-scale LOCAL CPU, not remote IO — this is the low end of the problem class (variant D loop-invariant-not-hoisted), material only at very high split counts (~100k), and it is legacy-parity behavior (old IcebergScanNode.createIcebergSplit recomputed per split too). Per the audit rules a partial/absent mitigation means CONFIRMED, but this should rank near the bottom of the findings list.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 628, + "note": "Eager path: for (FileScanTask task : plan.tasks) -> buildRangeForTask per sliced task; streaming path does the same at :516-518. No dedup by DataFile or partition in either loop." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1670, + "note": "TableScanUtil.splitFiles(withNoopClose(fileScanTasks), targetSplitSize) — one data file becomes ceil(fileSize/targetSplitSize) tasks sharing one DataFile; forced-size variant at :1657. Defaults: 64MB max / 32MB initial split size (:135-136), so 2-8 slices per typical parquet file." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1056, + "note": "buildRange recomputes getPartitionDataJson (:1056), getIdentityPartitionInfoMap (:1060), LinkedHashMap reorder (:1061-1067), and buildDeleteFiles (:1117) per call; gated on partitioned && instanceof PartitionData (:1051). No memo field or map anywhere in the method or class." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 576, + "note": "Counter-evidence of partial hoisting: scan-level invariants orderedPartitionKeys/zone/partitioned/vendedToken/formatVersion computed once per planScanInternal (:575-587) — the per-scan tier IS hoisted; only the per-partition/per-file tier is not." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 201, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE = false — the only cache on this path is off by default AND orthogonal: the manifest-cache branch feeds the identical downstream buildRange (:1702-1703 javadoc: 'the downstream buildRange (T03-T07) is unchanged')." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 208, + "note": "getPartitionDataJson: per-call getPartitionValues (per-field transform dispatch + date/time/zone formatting, :363-388) + JsonUtil.mapper().writeValueAsString (:211). Mapper is a static singleton (mitigates mapper construction only); no result memo. findColumnName (:161) hits iceberg's per-Schema cached idToName index — O(1) after first call, mildly weakening the finder's 'two schema lookups' heaviness claim." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1129, + "note": "buildDeleteFiles allocates a fresh carrier list + convertDelete (normalizeUri + ByteBuffer bounds decode :1183-1193) per slice, even though all k slices of one file share the same cached task.deletes() list (:1090-1091 comment confirms the list itself is cached, no IO)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 475, + "note": "Generic node only consumes fileSplit.getPartitionValues() (normalizeColumnsFromPath per split) — no upstream recompute, but also no dedup layer; per-range thrift bytes are inherent wire cost, not part of this finding." + } + ], + "corrected_multiplicity": "per-split (per sliced FileScanTask): N_splits = Σ_files ceil(fileSize/targetSplitSize), default target 64MB. Partition JSON + identity map are invariant per (specId, partition tuple) — reducible to O(distinct partitions), typically 10-1000; delete conversion is invariant per parent DataFile — reducible to O(files). Both currently run at O(splits) on the eager (:628) AND streaming (:516) paths.", + "mitigation_found": "Partial only: (1) scan-level invariants (orderedPartitionKeys/zone/partitioned/vendedToken/formatVersion) are already hoisted once per planScanInternal (:575-587) — the missing tier is (specId,partition) / per-DataFile; (2) iceberg SDK internals blunt per-call cost: Schema.findColumnName uses a per-Schema cached idToName index, JsonUtil.mapper() is a static singleton, task.deletes() is a cached list — but none prevent the repeated Jackson serialization/formatting/allocation; (3) IcebergManifestCache is off-by-default (:201) and orthogonal (caches manifest reads, buildRange unchanged :1702-1703). No memo keyed on partition data or DataFile exists in the provider, IcebergPartitionUtils, fe-connector-cache, fe-connector-metastore-iceberg, or fe-core's consuming node.", + "impact_estimate": "O(splits) x local-cpu (µs-scale Jackson JSON + date/timestamp zone formatting + per-delete decode/alloc; NO remote IO) x broad trigger: all partitioned iceberg tables for the JSON/identity part (gate :1051), any v2+ MOR table for delete conversion; both eager and streaming paths. ~5-15µs/split worst case (timestamp partitions + several deletes) -> roughly 0.5-1.5s wasted planning CPU + GC pressure at 100k splits, tens of ms at 10k, noise below 1k. Legacy-parity (old IcebergScanNode recomputed per split too), so no regression vs master — low-rank finding.", + "fix_direction": "memoize (per-planning-pass, no cross-query cache): a small HashMap<(specId, partitionData), {json, orderedValues}> local to planScanInternal / the streaming source, consulted lazily in buildRange (PartitionData implements equals/hashCode in iceberg, or key on specId + dataFile.path() prefix); plus reuse the converted delete-carrier list across the k slices of one parent task (key on identity of the cached task.deletes() list or on dataFile.path()). Pure hoist within existing plumbing — no SPI/thrift change needed." + } + }, + { + "title": "v3 rewritable-delete stash converts the same delete files to thrift twice per split and re-puts an identical supply for every slice of a file", + "variant": "D-loop-invariant-not-hoisted", + "heavy_op": "IcebergScanRange.DeleteFile.toThrift() list build in rewritableDeleteDescs() at plan time, then AGAIN per range in populateRangeParams; stash.accumulate ConcurrentHashMap put per split with an identical (path -> descs) pair", + "multiplicity": "per-split x per-delete-file, twice: buildRangeForTask loop (IcebergScanPlanProvider.java:627-637 -> :701-703) at plan time, then FileQueryScanNode.splitToScanRange (fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java:486) -> setScanParams -> populateRangeParams per range at thrift-assembly time", + "entry_chain": "IcebergScanPlanProvider.buildRangeForTask (fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java:701-703) -> IcebergScanRange.rewritableDeleteDescs (fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java:302-313) -> DeleteFile.toThrift (IcebergScanRange.java:681-704) -> IcebergRewritableDeleteStash.accumulate (fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java:101-116); second conversion: PluginDrivenScanNode.setScanParams (fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java:1673) -> IcebergScanRange.populateRangeParams (IcebergScanRange.java:413-417)", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 302 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 701 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java", + "line": 115 + } + ], + "why_heavy": "For a format-version>=3 scan every split builds a full TIcebergDeleteFileDesc list once for the stash (equality-filtered) and a second, overlapping list later for the wire; the k slices of one data file redundantly rebuild and re-put byte-identical supplies (the stash comment itself notes the put is 'idempotent' — i.e. k-1 of them are wasted conversions), and stashing happens even for plain SELECTs that never consume the supply (leaked entry aged out by TTL). CPU + allocation only; no remote IO.", + "gate": "Only format-version >= 3 tables with the stash wired (stashRewritableDeletes, IcebergScanPlanProvider.java:613) — plan-time conversion runs for every v3 SELECT, not just DML", + "est_impact": "2x thrift conversion of every delete desc + k redundant map puts per file; ~µs x N_splits x deletes — noticeable (100ms-1s) only on delete-heavy v3 tables with 10k+ splits", + "confidence": "high", + "lens": "per-split-serialization", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Every mechanical claim re-derives from the code. (1) Plan time: IcebergScanPlanProvider.planScanInternal gates the stash side-effect at :613 on only `rewritableDeleteStash != null && formatVersion >= 3` — no DML check — and the stash is unconditionally instantiated in production (IcebergConnector.java:167); v3 is forced onto this eager path because streamingSplitEstimate returns -1 for v3 (:416-418). The :628 loop iterates FileScanTasks produced by TableScanUtil.splitFiles (:1657), i.e. per byte-slice, so a data file in k slices reaches buildRangeForTask k times; :701-703 calls rewritableDeleteDescs() (IcebergScanRange.java:302-313 — fresh ArrayList + DeleteFile.toThrift per non-equality delete, no caching) and accumulate() (IcebergRewritableDeleteStash.java:115 — CHM put) once PER SPLIT, producing k byte-identical (path -> descs) puts per file; the stash's own javadoc (:97-99) concedes the put is merely \"idempotent\". Plain SELECTs on v3 tables pay this and leak the entry to a 300s TTL sweep (:51-57, :62-64). (2) Assembly time: FileQueryScanNode per-split loop (:434, streaming :379) -> splitToScanRange (:461) -> setScanParams (:486) -> PluginDrivenScanNode.setScanParams (:1662) -> populateRangeParams (:1673) -> IcebergScanRange.populateRangeParams :413-417 re-converts ALL deletes via fresh toThrift per range. So each non-equality delete desc is converted exactly twice per split with no sharing at any layer. Heaviness holds as claimed — the candidate honestly framed it as CPU+allocation only: toThrift (:681-704) is a small struct build (path ref + up to ~6 boxed fields; the equality field-id copy is filtered out of the stash path at :308), task.deletes() is a cached list (provider :1090, no IO). Nothing refutes: no memoization, not once-per-query, not dead code, incurred on every production v3 scan. Caveat on severity, not validity: this is a textbook variant-D loop-invariant-not-hoisted, but it sits at the low edge of the problem class's heavy-op bar (small thrift structs, not schema-sized), and the candidate's 100ms-1s ceiling is roughly an order of magnitude high.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 613, + "note": "stashRewritableDeletes = rewritableDeleteStash != null && formatVersion >= 3 — no DML gate; plain v3 SELECTs pay the plan-time conversion" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 628, + "note": "per-FileScanTask loop; tasks are byte-slices from TableScanUtil.splitFiles(scan.planFiles(), fileSplitSize) at :1657, so the loop body runs per split, k times per data file" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 701, + "note": "buildRangeForTask calls range.rewritableDeleteDescs() + stash.accumulate PER SPLIT (:701-703); conversion result is invariant across slices of one file" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 416, + "note": "streamingSplitEstimate returns -1 for formatVersion >= 3, so every v3 scan takes the eager path where the stash side-effect actually runs (cost always incurred)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 302, + "note": "rewritableDeleteDescs(): fresh ArrayList + toThrift() per non-equality delete on every call, no memoization (:302-313); equality deletes filtered at :308" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 413, + "note": "populateRangeParams v2+ branch re-converts ALL deleteFiles via fresh toThrift() per range (:413-417) — the second conversion of the same delete files, at thrift-assembly time" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 681, + "note": "DeleteFile.toThrift() = small struct build (path reference + up to ~6 boxed scalars, fieldIds copy only for equality deletes which the stash path excludes) — sub-microsecond, no IO; bounds the per-op cost" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java", + "line": 115, + "note": "entry.sets.put(rawDataFilePath, deleteDescs) — k identical puts for a file in k slices; javadoc :97-99 admits splits carry identical lists and the put is 'idempotent' (= k-1 wasted); :51-57 confirms plain-SELECT entries leak until the 300s TTL sweep" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 167, + "note": "rewritableDeleteStash unconditionally instantiated per connector — the stash!=null half of the gate is always true in production" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 486, + "note": "per-split loop (:434, streaming :379) -> splitToScanRange (:461) -> setScanParams(rangeDesc, fileSplit) at :486 — the fe-core hop that drives the second per-range conversion" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1673, + "note": "setScanParams (:1662) -> scanRange.populateRangeParams(tableFormatFileDesc, rangeDesc) — per-range entry into the wire-side toThrift loop" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1090, + "note": "task.deletes() is a cached list (comment: 'no extra I/O') — confirms the redundant work is pure CPU/allocation, never remote IO" + } + ], + "corrected_multiplicity": "As claimed: per-split x per-non-equality-delete-file, twice (plan-time stash at IcebergScanPlanProvider.java:701-703, wire-time at IcebergScanRange.java:413-417), plus k-1 redundant byte-identical stash conversions+puts per data file split into k slices. Gate: format-version >= 3 tables only (stash itself is always wired), incurred by every v3 statement including plain SELECT; v3 never takes the streaming path so there is no bypass.", + "mitigation_found": "None. rewritableDeleteDescs() and DeleteFile.toThrift() build fresh objects on every call (no cached field); accumulate() does not check-before-convert (the conversion cost is already paid by the time the idempotent put deduplicates); nothing distinguishes DML from SELECT at scan-plan time, so non-consuming statements pay the conversion and leak the entry to the 300s TTL sweep.", + "fix_direction": "Hoist the stash conversion from per-split to per-file: cheapest is to skip accumulate for non-first slices (e.g. only when task.start() == 0, or consult entry.sets.containsKey(rawPath) BEFORE calling rewritableDeleteDescs() so the toThrift work is skipped, not just the put). A deeper fix stashes the lightweight DeleteFile carriers (already built once per range for the wire) and defers toThrift to retrieveAndRemove — one conversion per file, executed only by the DML write provider, making plain SELECTs pay nothing but a map put. Optionally share one wire-side desc list across slices of the same file in populateRangeParams (thrift serialization does not mutate), though that touches the inherent wire path and is a smaller win.", + "impact_estimate": "Lower than the candidate's 100ms-1s: extra work is N_splits x D_nonEq sub-microsecond struct builds + N_splits CHM puts, all in-memory. A delete-heavy v3 table at 10k splits x ~10 non-eq deletes/file is ~100k redundant toThrift calls + 100k puts, i.e. single-digit-to-tens of ms of planning CPU plus allocation/GC pressure — real but dwarfed by the same query's inherent planFiles remote IO. Severity: low; pattern validity (variant D, no hoist at any layer): high." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The candidate's mechanics are all verified and no existing mechanism prevents the repeated cost in the default configuration. (1) The plan loop is per-slice, not per-file: plan.tasks come from TableScanUtil.splitFiles (IcebergScanPlanProvider.java:1657/1670/1781), and buildRangeForTask (:701-703) calls range.rewritableDeleteDescs() + stash.accumulate for EVERY slice; slices of one data file share task.file()/task.deletes(), so k-1 of the k conversions+puts per file are byte-identical waste (the stash put at IcebergRewritableDeleteStash.java:115 is a plain overwrite of the same key with a freshly-allocated identical list). (2) Neither rewritableDeleteDescs (IcebergScanRange.java:302-313) nor DeleteFile.toThrift (:681-704) memoizes; the fe-connector-cache manifest cache and snapshot caches cover manifest/table loads, not thrift conversion — nothing at any layer caches these descs. (3) The stash conversion runs for plain SELECTs too: the stash is constructed unconditionally (IcebergConnector.java:167) and wired into every scan provider (:589-591); the only gate is formatVersion>=3 (IcebergScanPlanProvider.java:613); and v3 is explicitly barred from the streaming path (streamingSplitEstimate returns -1 at :416-418), so every v3 scan of any size takes the eager loop with the side-effect. (4) The second conversion at populateRangeParams (IcebergScanRange.java:413-417, reached via FileQueryScanNode.splitToScanRange:486 -> PluginDrivenScanNode.setScanParams:1673) is verified, but that per-range wire build is inherent to the BE thrift contract (legacy setIcebergParams did the same), so the NEW redundancy is exactly the stash-side conversion + per-slice re-put. Partial mitigations exist and narrow the trigger without removing it: accumulate no-ops on empty lists (stash :102-105) and rewritableDeleteDescs short-circuits on no-delete files (:303-305), so only v3 files actually carrying non-equality deletes (DVs/position deletes) pay; COUNT(*) pushdown returns before the stash setup (:594-600); the 300s TTL sweep (:64,:131-134) bounds the SELECT leak in memory only. Crucially there is NO remote IO: task.deletes() is an SDK-cached list (provider :1090-1091 comment), so this is pure local CPU/allocation — real and unmitigated, but micro-scale per occurrence, and the candidate's 100ms-1s upper bound is over-stated for typical v3 tables (<=1 DV per file).", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 613, + "note": "Gate is only 'stash non-null && formatVersion >= 3' — no DML-statement gate; plain SELECTs on v3 tables pay the stash conversion (comment at :610-612 admits non-DML scans just leak an entry)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 701, + "note": "buildRangeForTask runs stash.accumulate(range.getOriginalPath(), range.rewritableDeleteDescs()) per FileScanTask; tasks are slice-level (TableScanUtil.splitFiles at :1657/:1670/:1781), so k slices of one file rebuild+re-put identical desc lists." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 302, + "note": "rewritableDeleteDescs() builds a fresh List on every call (equality-filtered) — no cached field, no lazy supplier; short-circuits only when deleteFiles is empty (:303-305, partial mitigation)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 413, + "note": "populateRangeParams rebuilds ALL delete descs (including equality) per range at thrift-assembly time — the second conversion; this one is inherent to the per-range BE wire contract (legacy parity), so the stash conversion is the added redundancy." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 681, + "note": "DeleteFile.toThrift() allocates a fresh TIcebergDeleteFileDesc each call (~7 field sets, copies fieldIds list for equality deletes) — no memoization at this layer either." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java", + "line": 115, + "note": "entry.sets.put(rawDataFilePath, deleteDescs) is a plain overwrite — the k-th slice of a file replaces an identical list; no presence check that would let callers skip the conversion. Empty-list no-op at :102-105 is the only (partial) input filter; TTL sweep (:64, :131-134) bounds SELECT-leak memory only, not CPU." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 167, + "note": "Stash instantiated unconditionally per catalog and passed to every scan provider (:589-591) and write provider (:600-602) — in production rewritableDeleteStash is never null, so the :613 gate reduces to formatVersion>=3." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 416, + "note": "streamingSplitEstimate returns -1 for formatVersion>=3 — v3 tables of ANY split count are forced onto the eager planScan path where the stash side-effect runs (the streaming path's stashRewritableDeletes=false is not a mitigation for v3)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1090, + "note": "Comment confirms task.deletes() is an SDK-cached list (no extra I/O) — the entire finding is local CPU/allocation, zero remote IO." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1673, + "note": "setScanParams delegates to scanRange.populateRangeParams per range, reached per split from FileQueryScanNode.splitToScanRange (FileQueryScanNode.java:486) — second-conversion chain verified as cited." + } + ], + "corrected_multiplicity": "per-split (slice) x per-non-equality-delete-file for the redundant stash conversion + put (k-1 of k puts per multi-slice file are byte-identical waste, and 100% are waste on non-DML statements); the populateRangeParams conversion is per-range x per-delete-file but INHERENT (BE wire contract, legacy parity) — the net amplification is 1 extra conversion pass per slice, not a true 2x of new cost.", + "mitigation_found": "Partial only: (a) rewritableDeleteDescs short-circuits on no-delete files (IcebergScanRange.java:303-305) and accumulate no-ops on empty lists (IcebergRewritableDeleteStash.java:102-105) — narrows trigger to v3 files with DVs/position deletes; (b) COUNT(*) pushdown returns before stash setup (IcebergScanPlanProvider.java:594-600); (c) 300s TTL sweep bounds the plain-SELECT leak in MEMORY only (stash :64, :131-134), not CPU. NO memoization of the desc conversion at any layer (range field, toThrift, fe-connector-cache), no per-file dedupe before the per-slice put, and no DML gate — the stash is wired unconditionally (IcebergConnector.java:167, :589-591) and v3 cannot escape to the stash-inert streaming path (:416-418).", + "impact_estimate": "O(splits x non-eq-deletes-per-split) x local-cpu/allocation (zero remote IO) x narrow trigger: only format-version>=3 iceberg tables whose scanned files carry deletion vectors or position deletes, on EVERY statement type (SELECT included), all catalog types. Per occurrence ~100-300ns (one small thrift object + field sets) plus ~50-100ns CHM put per slice. v3 spec caps DVs at 1 per data file, so typical extra cost at 10k splits is ~1-5ms; reaching the candidate's 100ms-1s requires ~100k slices x many v2-upgrade-leftover position deletes per file. Real, unmitigated D-variant waste, but 1-2 orders below the candidate's upper estimate and dwarfed by planFiles manifest IO on the same path — a low-severity finding worth a cheap fix, not a planning-latency hazard.", + "fix_direction": "Hoist + memoize + gate, in descending value: (1) hoist the conversion to per-file — in the planScanInternal loop (or inside accumulate) skip work when the queryId already has the raw path (presence check BEFORE calling rewritableDeleteDescs), or memo (dataFile path -> descs) across slices of the same file; (2) reuse one conversion — have IcebergScanRange convert deleteFiles to thrift descs once (lazy cached field) and let both rewritableDeleteDescs (filter view) and populateRangeParams share it, halving conversions per slice; (3) gate stashing on a DML signal threaded through ConnectorSession/handle if available, eliminating all stash work for plain SELECTs (currently 100% waste there). No thrift schema change needed." + } + }, + { + "title": "Generic per-split assembly redoes loop-invariant path parsing and builds columns-from-path that the iceberg range immediately discards", + "variant": "D-loop-invariant-not-hoisted", + "heavy_op": "LocationPath.of(pathStr) (URLEncoder.encode + String.replace + URI.create) per split on an already-normalized path; new Hadoop Path parse in toStorageLocation() per split; parent-built columnsFromPath lists unset+rebuilt by populateRangeParams; new IcebergScanPlanProvider allocation + TCCL swap per split in getFileCompressType", + "multiplicity": "per-split, in the generic node: PluginDrivenSplit ctor per range (queue pump at PluginDrivenScanNode.java:1640 / getSplits), then FileQueryScanNode.splitToScanRange per split (loop at FileQueryScanNode.java:434-436 and the batch-mode SplitAssignment callback :379)", + "entry_chain": "FileQueryScanNode.createScanRangeLocations (fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java:435) -> splitToScanRange (:461): normalizeColumnsFromPath (:478) + createFileRangeDesc columnsFromPath (:566-570, discarded by IcebergScanRange.populateRangeParams unset at fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java:435-437) + getFileCompressType (:482 -> PluginDrivenScanNode.java:602-610 -> resolveScanProvider :224-226 -> IcebergConnector.getScanPlanProvider fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java:583-592 new provider per call) + rangeDesc.setPath via toStorageLocation (:573 -> LocationPath.java:404-406); split creation: PluginDrivenSplit.buildPath (fe/fe-core/src/main/java/org/apache/doris/datasource/split/PluginDrivenSplit.java:76-79 -> LocationPath.of(String) LocationPath.java:163-169, encodedLocation :346-354)", + "files": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/split/PluginDrivenSplit.java", + "line": 78 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 480 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 605 + } + ], + "why_heavy": "Four independent µs-scale per-split costs on inputs that repeat across the k slices of one file: (a) the connector already normalized the path (normalizeUri) yet PluginDrivenSplit re-parses it through URLEncoder+URI.create; (b) splitToScanRange re-parses it a third time via new Hadoop Path; (c) the parent materializes columns-from-path value/isNull lists that IcebergScanRange.populateRangeParams unconditionally unsets and rebuilds from its own partitionValues map — dead work for every iceberg split; (d) getFileCompressType allocates a fresh IcebergScanPlanProvider and swaps the TCCL per split only to run the identity adjustFileCompressType. (Verified NOT a problem: the per-split getFileFormatType at FileQueryScanNode:485 is memoized through cachedPropertiesResult, PluginDrivenScanNode.java:1776-1823.)", + "gate": "none (every plugin-driven scan)", + "est_impact": "~3-8µs x N_splits: a few hundred ms of planning CPU + garbage at 100k splits; invisible below 10k splits", + "confidence": "high", + "lens": "per-split-serialization", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "All four per-split mechanisms re-derived from code exactly as claimed, and the multiplicity holds on every enumeration path (eager, batch, streaming). (a) The iceberg provider normalizes each path via normalizeUri before emitting the range, yet PluginDrivenSplit.buildPath re-parses it with LocationPath.of(String) = URLEncoder.encode + 2x String.replace + URI.create per split — while LocationPath.ofDirect (the documented no-parse fast ctor) sits unused on this path. (b) FileQueryScanNode:573 parses the same string a third time via new Hadoop Path in toStorageLocation() just to toString() it back. (c) For identity-partitioned tables the provider emits path_partition_keys (IcebergScanPlanProvider:1325), so the parent builds columnsFromPath/isNull lists (normalizeColumnsFromPath :478, thrift set :566-569) that IcebergScanRange.populateRangeParams unconditionally unsets (:433-437) and rebuilds from its own partitionValues map — built-then-discarded work every split. (d) getFileCompressType per split resolves the provider via the SPI default (Connector.java:85-86 delegates to no-arg; IcebergConnector has no handle override), and IcebergConnector.getScanPlanProvider() constructs a fresh IcebergScanPlanProvider per call (ctor = field assigns + a new ConcurrentHashMap) plus two TCCL swaps in onPluginClassLoader, all to run the identity adjustFileCompressType default (ConnectorScanPlanProvider.java:125, no iceberg override). Nothing collapses: no memoization exists for any component; none is once-per-query; the cost is incurred on every plugin-driven iceberg scan. The finding's own exclusion is also correct: getFileFormatType at :485 IS memoized through cachedPropertiesResult (PluginDrivenScanNode:1776-1823). Severity caveat: heaviness is exactly the claimed us-scale local CPU + allocation (no remote IO), i.e. the low end of the problem class (variant D per the class doc: per-iteration string parsing / thrift-substructure building); impact is invisible at small split counts but the streaming path explicitly targets million-file scans (:1637), where the ceiling is seconds of planning CPU plus tens of millions of allocations — 'as claimed or worse'.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 434, + "note": "Eager per-split loop calling splitToScanRange (:435-436); batch/streaming path registers this::splitToScanRange as the per-split SplitAssignment callback at :379." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 478, + "note": "normalizeColumnsFromPath per split; createFileRangeDesc sets columnsFromPath/keys/isNull at :566-569 when pathPartitionKeys non-empty; setPath via toStorageLocation at :573; getFileCompressType at :482; memoized getFileFormatType at :485." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/split/PluginDrivenSplit.java", + "line": 76, + "note": "buildPath -> LocationPath.of(pathStr) per split (:76-78); buildPartitionValues allocates a fresh List per split (:81-94); one PluginDrivenSplit per range at PluginDrivenScanNode :1248 (eager), :1567 (batch), :1640 (streaming pump)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java", + "line": 163, + "note": "of(String) = extractScheme + encodedLocation + URI.create; encodedLocation (:346-354) = URLEncoder.encode + two full-string replaces; toStorageLocation (:404-405) = new Hadoop Path(normalizedLocation), the third parse; ofDirect (:215-219) is the existing no-parse ctor, unused on this path." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 874, + "note": "Connector already normalizes every data path via normalizeUri (:874, :1105) before building the range; emits path_partition_keys for identity-partitioned tables at :1325 (gate for the dead columns-from-path component); provider ctor is field-assign-only plus a fresh ConcurrentHashMap (:231, :261-270)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 435, + "note": "populateRangeParams unconditionally unsets the parent-built columnsFromPath/keys/isNull (:433-437) and rebuilds them from its own partitionValues map (:438-451) — parent work discarded every split." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 602, + "note": "getFileCompressType per split: resolveScanProvider (:605 -> :224-226) then onPluginClassLoader two-TCCL-swap wrapper (:633-641) around identity adjustFileCompressType; memoization of scan-node properties confirmed at :1776-1823 (excludes getFileFormatType from the finding, as the candidate itself stated)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 583, + "note": "getScanPlanProvider() builds a fresh IcebergScanPlanProvider per call (:583-592); no override of the handle-taking overload, so Connector.java:85-86 SPI default routes every resolveScanProvider() call here — one new provider object per split via getFileCompressType." + }, + { + "file": "fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java", + "line": 125, + "note": "adjustFileCompressType is the identity default and fe-connector-iceberg does not override it — the per-split provider allocation + TCCL swap buys a no-op for iceberg." + } + ], + "corrected_multiplicity": "per-split, exactly as claimed, on all three enumeration paths: eager getSplits (PluginDrivenScanNode:1248 + FileQueryScanNode:434-436), approximate-batch (:1567 + SplitAssignment callback FileQueryScanNode:379), and streaming (:1640 + same callback). Within the triple-parse component, the redundancy factor is 2-3x per split on the same string (connector normalizeUri -> LocationPath.of -> new Hadoop Path), and additionally k-x across the k slices of one large file; the provider-allocation + TCCL component is fully loop-invariant (same logical provider, identity adjust) yet re-done per split.", + "fix_direction": "Four independent hoists, all engine-side or connector-boundary, no behavior change: (1) resolve the scan provider once per scan node (a field, as startStreamingSplit already does at :1628) instead of per split in getFileCompressType — or have IcebergConnector cache its provider instance; (2) let PluginDrivenSplit use LocationPath.ofDirect (or a cheap scheme/authority extraction) since the connector guarantees a normalized URI — the URLEncoder+URI.create round-trip is only needed for un-normalized user paths; (3) setPath from the normalizedLocation string directly instead of the new Path(...).toString() identity round-trip at FileQueryScanNode:573; (4) skip the parent columns-from-path assembly (normalizeColumnsFromPath + thrift set) when the split's connector rewrites them in populateRangeParams — e.g. a boolean on ConnectorScanRange or on the split, mirroring the existing isPartitionBearing flag.", + "impact_estimate": "Low severity, real but only at scale: roughly 2-6us extra CPU plus ~10 short-lived allocations per split (URLEncoder buffer, URI, Hadoop Path, 2-5 ArrayLists, provider + ConcurrentHashMap). Invisible below ~10k splits; ~0.2-0.6s planning CPU + GC pressure at 100k splits; seconds at the million-file scans the streaming path is explicitly built for (PluginDrivenScanNode:1637). No remote IO involved — this is the low end of the problem class (variant D), not a planFiles-style amplification.", + "mitigation_found": "Partial mitigations exist but none covers the four reported components: getFileFormatType at FileQueryScanNode:485 is memoized via cachedPropertiesResult (PluginDrivenScanNode:1776-1823) — correctly excluded by the candidate itself; LocationPath.ofDirect (LocationPath.java:215-219) is the codebase's own recognition of the parse cost but is not used by PluginDrivenSplit; startStreamingSplit hoists resolveScanProvider once for streamSplits (:1628) but the per-split getFileCompressType path does not reuse it. Component (c) is gated: dead columns-from-path work occurs only for identity-partitioned tables (path_partition_keys emitted at IcebergScanPlanProvider:1325); unpartitioned tables degenerate to near-no-ops." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "Mitigation hunt result: only one of the four sub-claims has an existing mitigation, and the finder already excluded it. (1) MITIGATED (as the finder noted): the per-split getFileFormatType at FileQueryScanNode:485 routes through PluginDrivenScanNode:527-534 -> getOrLoadScanNodeProperties:1815-1823 -> getOrLoadPropertiesResult:1776-1810, which caches in cachedPropertiesResult/scanNodeProperties fields — the connector round trip runs once per scan, not per split. (2) NOT mitigated: PluginDrivenSplit.buildPath (PluginDrivenSplit.java:76-79) calls LocationPath.of(String) (LocationPath.java:163-169) per range at all three split-materialization sites (eager :1248, partition-batch :1567, streaming pump :1640); of(String) runs encodedLocation (URLEncoder.encode over the full path + two String.replace, LocationPath.java:346-354) + URI.create per call, on a path the connector ALREADY normalized (IcebergScanPlanProvider.normalizeUri:1222, applied at .path() build sites :874/:1105/:1157). Telling detail: LocationPath ships purpose-built mitigations — ofDirect (:215-220) and ofWithCache (:232+, javadoc: 'optimized for batch processing where many files share the same bucket/prefix') — but grep shows ZERO callers in fe-core main; the mitigation exists and is not wired to this path. (3) NOT mitigated: splitToScanRange re-parses the same string a third time via new Hadoop Path in toStorageLocation (LocationPath.java:404-406, called at FileQueryScanNode:573) and, for HDFS location type, a FOURTH time via getPath() (new Path, LocationPath.java:435-437) just to derive fsName (:574-576) — fsName is identical for every split sharing a filesystem, loop-invariant, never hoisted. (4) NOT mitigated (but gated): the parent builds columns-from-path value/isNull lists (normalizeColumnsFromPath FileQueryScanNode:478, FilePartitionUtils.java:158-174) and sets three thrift fields (:566-570) that IcebergScanRange.populateRangeParams unconditionally unsets and rebuilds from its own partitionValues map (IcebergScanRange.java:435-451). Partial natural mitigation: the thrift set/unset dead work only materializes when path_partition_keys is non-empty, i.e. identity-partitioned iceberg tables (IcebergScanPlanProvider.java:1325); unpartitioned tables hit the empty fast path (FilePartitionUtils.java:159-161). (5) NOT mitigated: getFileCompressType per split (FileQueryScanNode:482 -> PluginDrivenScanNode:602-610) calls resolveScanProvider (:224-226) -> Connector.java:85 default -> IcebergConnector.getScanPlanProvider (:583-592) which constructs a FRESH IcebergScanPlanProvider per call, plus a TCCL swap (onPluginClassLoader :633-641). No field in PluginDrivenScanNode caches the resolved provider despite 10+ resolveScanProvider call sites. The ctor is 5 field assignments (IcebergScanPlanProvider.java:261-270, catalog is behind a lazy resolver), so this is ~100ns of allocation per split — real, unmitigated, but the smallest item. Severity caveat (honest scrutiny): every unmitigated item is LOCAL CPU at micro-scale — no remote IO, no O(table) compute. This is a genuine D-variant (loop-invariant not hoisted, no cache at any layer, default config, every plugin-driven iceberg scan) but at the bottom of the severity range for this problem class: the per-split parse quadruple-up is the dominant term and the whole bundle is ~3-8us/split, invisible below ~10k splits and a few hundred ms of planner CPU + GC pressure at 100k splits, where manifest IO and thrift serialization likely still dominate. CONFIRMED per the rubric (no mitigation / partial mitigation), with impact at the low end of the finder's own estimate.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/split/PluginDrivenSplit.java", + "line": 78, + "note": "buildPath -> LocationPath.of(pathStr) per constructed split; ctor sites at PluginDrivenScanNode.java:1248 (eager), :1567 (partition-batch), :1640 (streaming pump) — verified all three" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java", + "line": 163, + "note": "of(String) = encodedLocation (URLEncoder.encode + 2x String.replace, lines 346-354) + URI.create per call; no memoization" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java", + "line": 232, + "note": "MITIGATION BUILT BUT UNUSED: ofWithCache (and ofDirect :215-220) exist precisely for 'many files share the same bucket/prefix' — grep finds zero callers in fe-core main" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1222, + "note": "connector already normalizes the path (normalizeUri, applied at .path() builders :874/:1105/:1157) before fe-core re-parses it — confirms the redundancy claim" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 573, + "note": "per-split third parse: fileSplit.getPath().toStorageLocation() = new Hadoop Path (LocationPath.java:404-406); lines 574-576 add a FOURTH parse (getPath() = new Path, LocationPath.java:435-437) to derive loop-invariant fsName for HDFS" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 605, + "note": "getFileCompressType per split: resolveScanProvider (:224-226) -> fresh IcebergScanPlanProvider per call (IcebergConnector.java:583-592, trivial 5-field ctor IcebergScanPlanProvider.java:261-270) + TCCL swap (:633-641); no node field caches the provider" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 435, + "note": "populateRangeParams unconditionally unsets the parent-built columnsFromPath/Keys/IsNull (:435-437) and rebuilds from its own partitionValues map (:438-451) — parent work at FileQueryScanNode:478,:566-570 is dead for iceberg" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1325, + "note": "GATE on the columns-from-path dead work: path_partition_keys only set for identity-partitioned tables; unpartitioned tables skip the thrift set (FileQueryScanNode:566 keys-empty check) and hit FilePartitionUtils.java:159-161 empty fast path" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1777, + "note": "VERIFIED MITIGATION (finder's exclusion is correct): cachedPropertiesResult null-check memoizes getScanNodePropertiesResult; getFileFormatType (:527-534) and getPathPartitionKeys (:537-544) read the cached map, so the per-split setFormatType at FileQueryScanNode:485 costs one map lookup after the first call" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 434, + "note": "the per-split multiplicity site: eager loop :434-436 calling splitToScanRange (:461); batch mode wires the same splitToScanRange as the SplitAssignment callback (:379) — same per-split costs on pool threads" + } + ], + "mitigation_found": "Partial. (1) getFileFormatType IS memoized (PluginDrivenScanNode cachedPropertiesResult/scanNodeProperties, :1776-1823) — the finder already excluded it; verified correct. (2) The columns-from-path dead work is naturally gated to identity-partitioned iceberg tables (path_partition_keys set only at IcebergScanPlanProvider:1325; empty-keys tables skip the thrift set and hit FilePartitionUtils:159-161 fast path). (3) LocationPath ships ofDirect/ofWithCache fast-path factories designed exactly for shared-prefix batches (LocationPath.java:215-220, :232+) but they have zero callers in fe-core main — mitigation exists, not wired. Nothing mitigates the per-split of(String) re-parse, the 3rd/4th Hadoop Path parses in splitToScanRange, or the per-split provider allocation + TCCL swap in getFileCompressType.", + "corrected_multiplicity": "per-split (unchanged from candidate): LocationPath.of once per PluginDrivenSplit ctor at all three materialization sites (PluginDrivenScanNode:1248 eager, :1567 partition-batch, :1640 streaming); then per split in splitToScanRange (FileQueryScanNode:434-436 eager loop / :379 batch callback): 1 normalizeColumnsFromPath + 1-2 new Hadoop Path + 1 provider allocation + 1 TCCL swap. The columns-from-path thrift set/unset/rebuild sub-item is per-split but only on identity-partitioned tables.", + "impact_estimate": "O(splits) x local-cpu (no remote IO anywhere in the finding) x all plugin-driven iceberg scans in default config (columns-from-path sub-item: identity-partitioned tables only). Dominant term is the 3-4x redundant path parsing (~2-5us/split: URLEncoder+URI.create+1-2 Hadoop Path ctors); provider allocation + TCCL swap ~100-200ns/split; dead columns-from-path lists ~200-500ns/split on partitioned tables. Aggregate ~3-8us/split matches the finder: ~0.3-0.8s extra planner CPU + allocation churn at 100k splits, negligible below ~10k splits. Low severity within the problem class — the 'heavy op' element is weak (micro-scale local CPU), but the no-hoist/no-memoize and high-multiplicity elements are fully satisfied.", + "fix_direction": "hoist + memoize, four small cuts: (1) cache the resolved ConnectorScanPlanProvider in a PluginDrivenScanNode field (resolveScanProvider is called 10+ times per scan and per split via getFileCompressType; the javadoc at :215-222 already states the provider cannot change across a scan); (2) pass-down: have PluginDrivenSplit/createFileRangeDesc reuse the connector-normalized path string directly for rangeDesc.setPath (skip the toStorageLocation Path round-trip) and derive fsName once per fsIdentifier (LocationPath already computes fsIdentifier), or wire the existing-but-unused LocationPath.ofWithCache per shared prefix; (3) skip the parent columns-from-path assembly when the connector rewrites it — a ConnectorScanRange capability flag (e.g. suppliesColumnsFromPath) letting splitToScanRange skip :478/:566-570, keeping fe-core source-agnostic; (4) hoist the identity adjustFileCompressType decision out of the loop: resolve the provider + check once, only enter onPluginClassLoader per split when the connector actually overrides." + } + }, + { + "title": "N-times-duplicated invariant payload: identical delete-file descriptors and partition JSON repeated in every range's thrift (per split of the same file, per file sharing a delete)", + "variant": "A-loop-amplification", + "heavy_op": "Serialization of the same TIcebergDeleteFileDesc list, partition_data_json string, and columns-from-path triple into every TFileRangeDesc of the plan / split-fetch RPC", + "multiplicity": "per-split payload: the k byte-slices of one data file each carry the file's FULL delete list + partition JSON (populateRangeParams per range via PluginDrivenScanNode.setScanParams:1662-1676); a shared delete file (equality/partition-wide position delete) is duplicated across ALL M data files it applies to => M x k desc copies in the serialized plan", + "entry_chain": "FileQueryScanNode.splitToScanRange (fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java:486-489, addToRanges) -> PluginDrivenScanNode.setScanParams (fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java:1673) -> IcebergScanRange.populateRangeParams (fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java:388-451: original path :392, partition JSON :396-397, delete descs :413-417, columns-from-path :438-451)", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 413 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 489 + } + ], + "why_heavy": "Payload-bytes finding (RPC/plan blowup), not CPU: each duplicated desc carries path + bounds + field-ids; on an equality-delete-heavy MOR table one delete path string (~100-200B) can appear tens of thousands of times in the serialized TFileScanRange set shipped to BEs (or streamed via split sources). CAVEAT — partially wire-contract-inherent: BE consumes delete files per range and TFileScanRangeParams has no params-level dedup dictionary today, and legacy IcebergScanNode emitted the same shape; the amplification is real but a fix requires a params-level (dedup-table + index) thrift change rather than an FE-side hoist. Reported per the lens's explicit payload mandate, ranked last for that reason.", + "gate": "Material only for v2+ tables with many merge-on-read delete files and/or high split counts", + "est_impact": "Plan-size / split-RPC bytes ∝ N_splits x avg-deletes-per-split; MBs of duplicated strings on large MOR scans (FE thrift build + BE parse time), zero effect on small or delete-free tables", + "confidence": "medium", + "lens": "per-split-serialization", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Both legs of the claim hold as stated. (1) Multiplicity: the payload is built per range on the planning hot path — FileQueryScanNode.splitToScanRange:486 -> PluginDrivenScanNode.setScanParams:1673 -> IcebergScanRange.populateRangeParams, which constructs a fresh TIcebergDeleteFileDesc per delete per range (:413-417) plus partition_data_json (:396-397) and columns-from-path (:438-451). Data files are byte-sliced by TableScanUtil.splitFiles (IcebergScanPlanProvider.java:463 streaming, :1657 eager) and each slice independently runs buildRange, which recomputes the per-file-invariant partition JSON (:1056), identity map (:1060) and rebuilds the delete-carrier list via buildDeleteFiles (:1117, :1129-1139) — SDK SplitScanTask.deletes() returns the parent file's full delete list, so all k slices of one file carry the full list, and the SDK DeleteFileIndex attaches a shared equality/partition-wide delete to every matching data file, giving the claimed M x k copies. (2) Heaviness: this is the 'large thrift structure serialization / loop-invariant thrift substructure rebuilt per iteration' category the problem-class doc explicitly lists (heavy-op bullet 3 and variant D); per-unit cost is small so it is material only on v2+ MOR tables with many deletes and/or high split counts — exactly the gate the candidate states. (3) No hoist/memoize anywhere: no per-file caching of carriers or JSON across slices, distinct thrift objects per range, and PlanNodes.thrift:339 defines delete_files as a per-range list with no params-level dedup dictionary in TFileScanRangeParams (:489). The candidate's own caveat is accurate and does not refute: the wire-bytes half is contract-inherent (BE consumes delete_files per range; legacy emitted the same shape — not a regression), but the FE-side CPU/heap half (per-slice recompute of partition JSON, identity map, normalizeUri, carrier conversion, fresh thrift objects) is genuinely un-hoisted and fixable FE-side today. Confirmed as claimed, appropriately ranked last.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 486, + "note": "splitToScanRange calls setScanParams per split inside the per-backend/per-split loop (:432-441); range added to plan at :489" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1673, + "note": "setScanParams delegates to scanRange.populateRangeParams for every range, no caching" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 413, + "note": "populateRangeParams builds a fresh List with delete.toThrift() per delete per range (:413-417); partition_data_json set per range (:396-397); columns-from-path rebuilt per range (:438-451); original_file_path (:392)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 463, + "note": "TableScanUtil.splitFiles(scan.planFiles(), sliceSize) byte-slices data files -> k slices per large file (streaming path; eager path identical at :1657)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1056, + "note": "buildRange recomputes getPartitionDataJson + identity map (:1060) per slice though invariant per data file" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1117, + "note": ".deleteFiles(buildDeleteFiles(task, vendedToken)) per slice; buildDeleteFiles (:1129-1139) converts task.deletes() (parent file's FULL cached list per SDK SplitScanTask, per comment :1090-1091) into fresh carriers with normalizeUri per delete per slice" + }, + { + "file": "gensrc/thrift/PlanNodes.thrift", + "line": 339, + "note": "delete_files is a per-range list inside TIcebergFileDesc; TFileScanRangeParams (:489) has no params-level dedup dictionary — wire duplication is contract-shaped" + } + ], + "corrected_multiplicity": "per-split (per byte-slice), exactly as claimed: M data files sharing a delete x k slices per file => M x k full delete-desc copies + k partition-JSON/columns-from-path copies per file, in both the eager plan and the streaming split source", + "fix_direction": "Two independent halves. (a) FE-side, no wire change: hoist per-file-invariant work across a file's k slices — compute partitionDataJson, identity map, and the converted delete-carrier list once per parent data file (keyed by the parent FileScanTask/DataFile) and share across its slice ranges; sharing the same TIcebergDeleteFileDesc instances across ranges also cuts FE heap/CPU (thrift serialization walks shared objects fine) but not wire bytes. (b) Wire bytes: requires a thrift contract change — a params-level delete-file dictionary (list at TFileScanRangeParams + per-range index references) plus a BE reader change; legacy emitted the same shape, so this is a protocol evolution, not a regression fix.", + "impact_estimate": "Zero on v1/delete-free tables. On equality-delete-heavy v2+ MOR scans with high split counts: plan/split-RPC bytes grow as N_splits x avg-deletes-per-split (each desc ~100-300B of path+bounds+field-ids) — potentially MBs of duplicated strings per plan; FE-side also pays per-slice JSON serialization + URI normalization + object churn that is per-file hoistable today. Second-order, not a planning-latency cliff like the canonical planFiles case.", + "mitigation_found": "None at any layer: no per-file memo of carriers/JSON across slices (buildRange recomputes per slice), no shared thrift instances across ranges (toThrift per range), no params-level dedup dictionary in the thrift contract. Gate only: material solely for v2+ merge-on-read tables with delete files and/or files larger than the slice size." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The duplication is real and unmitigated at every layer, in both split-delivery modes. Verified chain: FileQueryScanNode.splitToScanRange (per split) -> PluginDrivenScanNode.setScanParams:1673 -> IcebergScanRange.populateRangeParams, which builds a FRESH TIcebergDeleteFileDesc list (:413-417 via delete.toThrift() per call), partition JSON (:396-398), and columns-from-path (:438-451) into every TFileRangeDesc. Upstream, both the sync (planScan:631) and streaming (:517) paths call buildRangeForTask -> buildRange per FileScanTask, and TableScanUtil.splitFiles produces k byte-slice tasks per data file that share one task.deletes() list — yet buildRange recomputes partitionDataJson (:1056), the identity partition map (:1060), and buildDeleteFiles carrier objects with per-delete URI normalization (:1117, :1129-1139, :1156-1157) fresh for EVERY slice, with no memo keyed on data file, partition data, or delete path. Thrift contract confirmed: delete_files is a per-range list (PlanNodes.thrift:339 inside TIcebergFileDesc, carried in TFileRangeDesc.table_format_params:582); the params-level TFileScanRangeParams.table_format_params (:525) is commented Deprecated and holds no dedup dictionary. Batch mode does NOT dedup: SplitAssignment is constructed with the same this::splitToScanRange (FileQueryScanNode:379), so split-fetch RPCs re-serialize the identical per-range payload — batch mode only bounds FE heap and single-RPC size. Caveats that temper severity (finder already flagged them, I confirm): (1) NO remote IO is amplified — the SDK's task.deletes() is a cached shared list (comment :1090), so this is purely local-CPU + serialized-bytes; (2) per-range delete_files is the current BE wire contract (slices of one file can land on different BEs) and legacy IcebergScanNode emitted the same shape, so the byte-level duplication is contract-inherent, not an SPI-migration regression. But per my verdict rules, all found mechanisms are partial/orthogonal, so CONFIRMED — with the note that the FE-side CPU portion (per-slice partition-JSON re-serialization + carrier rebuild) is hoistable today without any wire change, and one under-called-out sub-item: partitionDataJson recompute hits ALL partitioned iceberg tables, not just MOR tables.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 486, + "note": "splitToScanRange calls setScanParams per split; :489 addToRanges — one TFileRangeDesc per split; :379 batch mode wires the SAME this::splitToScanRange into SplitAssignment, so streamed split-fetch RPCs carry identical per-range payloads (batch mode is not a dedup mitigation)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1673, + "note": "setScanParams (:1662-1676) delegates per-range thrift construction to scanRange.populateRangeParams; a new TTableFormatFileDesc per range." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java", + "line": 413, + "note": "populateRangeParams builds a fresh List per range (:413-417, delete.toThrift() per call — toThrift at :681 has no caching); partition JSON set per range :396-398; columns-from-path rebuilt :438-451. No dedup or shared-desc reuse." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 463, + "note": "Streaming path: TableScanUtil.splitFiles(scan.planFiles(), sliceSize) yields k byte-slice FileScanTasks per data file, each mapped independently via buildRangeForTask (:517); sync path identical (:631)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1056, + "note": "buildRange recomputes partitionDataJson (getPartitionDataJson, JSON serialization) and identity map (:1060) per FileScanTask — i.e., k times per data file for identical PartitionData, M*k times across files in one partition. Hits ALL partitioned iceberg tables, not just MOR." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1129, + "note": "buildDeleteFiles creates fresh carrier objects per slice (:1134-1137) with per-delete URI normalization in convertDelete (:1157); no memo keyed on delete path or data file. MITIGATION FOUND (partial): comment :1090 confirms task.deletes() is an SDK-cached shared list — no repeated remote IO, cost is local-CPU + bytes only." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1024, + "note": "planCountPushdown collapses to a single range — a real but narrow mitigation (bare COUNT(*) only). Manifest cache (:194 comment, default OFF) caches planning inputs, not output payload — irrelevant." + }, + { + "file": "gensrc/thrift/PlanNodes.thrift", + "line": 339, + "note": "delete_files is a per-range list inside TIcebergFileDesc, carried in TFileRangeDesc.table_format_params (:582); TIcebergDeleteFileDesc (:315-331) carries path + original_path strings + bounds + field_ids. Params-level TFileScanRangeParams.table_format_params (:525) is marked Deprecated — no params-level dedup dictionary exists in the wire contract." + } + ], + "corrected_multiplicity": "Per-range (= per byte-slice), O(splits): k slices of one data file each carry the file's full delete-desc list and partition JSON (k = fileSize/sliceSize, default slice ~= max_file_split_size); a shared equality/partition-wide delete file is copied into all M data files it applies to => M*k serialized desc copies per query. The partition-JSON/identity-map recompute is additionally M*k per partition on ALL partitioned iceberg tables (delete-free included), which the original finding understated.", + "mitigation_found": "Four mechanisms found, none sufficient: (1) Iceberg SDK task.deletes() is a cached list shared across slices (IcebergScanPlanProvider.java:1090 comment) — eliminates repeated remote IO, confining the finding to local-CPU + payload bytes; (2) batch/streaming split mode (auto for large scans) ships only a split-source ID in the plan and bounds FE heap, but SplitAssignment uses the same splitToScanRange (FileQueryScanNode.java:379) so total duplicated bytes across fetch RPCs are unchanged; (3) count-pushdown single-range collapse (:1021-1036) — bare COUNT(*) only; (4) IcebergManifestCache — default OFF and caches planning inputs, not thrift output. No layer dedups the M*k payload in the default configuration.", + "impact_estimate": "O(splits) x (local-cpu + serialized-bytes), zero remote IO. Delete-desc duplication: only v2+ merge-on-read tables with delete files — ~200-300B/desc x N_splits x avg-deletes-per-split (e.g. 10k splits x 5 deletes ~= 10-15MB of duplicated thrift across plan/split-fetch RPCs, FE build + BE parse CPU); tens of MB on equality-delete-heavy tables. Partition-JSON + identity-map recompute: ALL partitioned iceberg tables, k JSON serializations per file (pure FE CPU, small per call). Legacy-parity shape (old IcebergScanNode emitted the same), so not a regression; bounded impact, unlikely to dominate planning latency versus planFiles IO — consistent with the finder ranking it last.", + "fix_direction": "Two-tier. Tier 1 (FE-only, no wire change): per-planning-pass memoization — cache partitionDataJson + identity map keyed on (specId, PartitionData) and reuse one immutable DeleteFile-carrier list per data file across its k slices (slices already share task.deletes(); the carrier/URI-normalization rebuild is pure waste). Cuts the CPU amplification but not wire bytes (thrift has no structure sharing). Tier 2 (wire bytes, BE-coordinated thrift change): params-level dedup dictionary — a list at TFileScanRangeParams/scan level plus per-range list indices; requires BE reader changes, so it is a cross-team contract change, not an FE hoist." + } + }, + { + "title": "No Table-object cache at any layer: one planning pass performs 3-4 independent catalog.loadTable() remote loads while the 'table cache' TTL knob only caches a (snapshotId, schemaId) pair for beginQuerySnapshot", + "variant": "B-chain-redundancy", + "heavy_op": "IcebergCatalogOps.CatalogBackedIcebergCatalogOps.loadTable -> catalog.loadTable() (HMS getTable RPC + metadata.json FileIO read+parse / REST GET table; per load a full remote metadata fetch)", + "multiplicity": "k-times-per-query, k=3 minimum (isBatchMode->streamingSplitEstimate, getSplits->planScan, props compute) and k=4 with a WHERE clause (props computed twice, finding 2); +1 on latest-snapshot-cache miss (beginQuerySnapshot) and +1 more on the streaming path (streamSplits)", + "entry_chain": "hop1: FileQueryScanNode.createScanRangeLocations:376 isBatchMode -> PluginDrivenScanNode.computeBatchMode:1414 -> IcebergScanPlanProvider.streamingSplitEstimate:410 resolveTable; hop2: PluginDrivenScanNode.getSplits:1242 planScan -> IcebergScanPlanProvider.planScanInternal:562 resolveTable; hop3(+4): PluginDrivenScanNode.getOrLoadPropertiesResult:1801 -> IcebergScanPlanProvider.getScanNodeProperties:1300 resolveTable; all -> IcebergScanPlanProvider.resolveTable:1981-1989 -> IcebergCatalogOps.java:340-341 catalog.loadTable (no cache in fe-connector-metastore-iceberg providers, no iceberg CachingCatalog wrap in IcebergConnector.createCatalog:692-757). Only cache on the path: IcebergLatestSnapshotCache (wired IcebergConnector.java:188-189), read solely by IcebergConnectorMetadata.beginQuerySnapshot:1536-1545 and caching only the two ids", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java", + "line": 33 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 188 + } + ], + "why_heavy": "Each loadTable is a full remote metadata load: metastore RPC plus fetching and JSON-parsing metadata.json (can be MBs on snapshot-heavy tables; TableMetadataParser is also nontrivial CPU). IcebergLatestSnapshotCache's own javadoc says it 'restores the legacy IcebergExternalMetaCache table-cache semantics that the SPI cutover dropped' — but the legacy cache held the loaded Table OBJECT shared by all planning phases, whereas the restored projection saves only beginQuerySnapshot's load. meta.cache.iceberg.table.ttl-second (default 24h) therefore does NOT prevent 3-4 fresh remote loads per query. Side effect: each fresh loadTable creates a fresh FileIO instance, so even the SDK-level io.manifest.cache-enabled content cache (ManifestFiles caches keyed by FileIO identity, derived in IcebergCatalogFactory.appendManifestCacheProperties:119-133) can never hit across the phases.", + "gate": "none — every iceberg query on the plugin-driven path", + "est_impact": "3-4x metastore RPC + metadata.json fetch/parse per query vs the 1x a table-object cache would give; dominates planning latency for small/point queries and multiplies metastore load by ~3-4x at cluster QPS", + "confidence": "high", + "lens": "cache-audit", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Heaviness holds: resolveTable (IcebergScanPlanProvider:1981-1993) has no memo and delegates to CatalogBackedIcebergCatalogOps.loadTable (IcebergCatalogOps.java:340-341) = raw catalog.loadTable(), remote IO by the ground-truth cost model (HMS getTable RPC + metadata.json fetch/parse, or REST GET). No cache at any layer: no CachingCatalog wrap in IcebergConnector.createCatalog (692-757), zero CachingCatalog hits in fe-connector-iceberg/fe-connector-metastore-iceberg, and the bundled iceberg-core 1.6.1 CatalogUtil.class contains no CachingCatalog reference either. Multiplicity holds at the claimed minimum k=3 per query on defaults, traced end-to-end: (1) FileQueryScanNode.createScanRangeLocations:325 getFileFormatType -> getOrLoadScanNodeProperties -> getScanNodePropertiesResult (PluginDrivenScanNode:1801-1803; SPI default ConnectorScanPlanProvider.java:455-462 wraps getScanNodeProperties) -> resolveTable at IcebergScanPlanProvider:1300; (2) FileQueryScanNode:376 isBatchMode -> computeBatchMode (PluginDrivenScanNode:1414, gated only on hasSlots + enable_external_table_batch_mode default true, checked at IcebergScanPlanProvider:407 BEFORE the load) -> resolveTable at :410; (3) FileQueryScanNode:422 getSplits -> planScan (PluginDrivenScanNode:1242) -> planScanInternal resolveTable at :562, or on the streaming path streamSplits resolveTable at :452 which REPLACES planScan (startSplit:1498-1502). The only cache on the path, IcebergLatestSnapshotCache, stores exactly (snapshotId, schemaId) (lines 56-65) and is consumed solely by beginQuerySnapshot (IcebergConnectorMetadata:1536-1548), whose loader loadTable (line 1541) adds +1 on TTL miss — the meta.cache.iceberg.table.ttl-second knob covers none of the three planning loads. Two sub-claims corrected without collapsing the finding: the k=4-with-WHERE 'props computed twice' claim did NOT verify (convertPredicate's memo invalidation at PluginDrivenScanNode:796-797 runs before the first props compute per FileQueryScanNode:252-253 ordering; iceberg has no applyFilter override; no pre-finalize props reader found), and the '+1 more on the streaming path' is wrong as stated (streamSplits substitutes for planScan, so streaming is also k=3). Steady-state is therefore 3 redundant remote loads of loop-invariant information per iceberg query (4 on first query per TTL window; +1 more only under explicit time travel via resolveTimeTravel:1574), where a table-object cache or per-query hoist would give 1.", + "corrected_multiplicity": "k=3 per query on default settings (props compute, streamingSplitEstimate, and planScan-or-streamSplits each do one catalog.loadTable); +1 on IcebergLatestSnapshotCache TTL miss (beginQuerySnapshot loader); +1 under explicit time travel (resolveTimeTravel). NOT k=4 with WHERE (props are computed once — the memo invalidation in convertPredicate precedes the first compute), and the streaming path is also 3 (streamSplits replaces planScan, not adds). k drops to 2 only when enable_external_table_batch_mode=false (estimate gate at IcebergScanPlanProvider:407 fires before the load).", + "mitigation_found": "Only IcebergLatestSnapshotCache (value = (snapshotId, schemaId) pair, IcebergLatestSnapshotCache.java:56-65), read solely by IcebergConnectorMetadata.beginQuerySnapshot:1536-1548 — it does not cover any of the three planning-pass loads. PluginDrivenScanNode memoizes the props RESULT per node (cachedPropertiesResult:1776-1810) and isBatchMode (isBatchModeCache:1385-1390), which prevents per-split amplification but not the cross-phase 3x loadTable redundancy. No CachingCatalog wrap anywhere (verified in both connector modules and in the bundled iceberg-core 1.6.1 CatalogUtil.class).", + "impact_estimate": "3x remote metadata loads per iceberg query steady-state (4x on first query in a TTL window) vs the 1x a table-object cache or per-query hoist would give: each load is a metastore RPC / REST GET plus metadata.json fetch + JSON parse (MBs on snapshot-heavy tables). Dominates planning latency for small/point queries and multiplies metastore/REST-endpoint load ~3x at cluster QPS; also defeats cross-phase sharing of the SDK's FileIO-identity-keyed manifest content cache since every load creates a fresh FileIO.", + "fix_direction": "Hoist the Table object to once per planning pass: memoize the resolved (auth-wrapped) Table per query — e.g. keyed by (queryId, db.table, snapshot pin) on the provider (mirroring the existing queryId-keyed scanProfileStash/rewritableDeleteStash pattern) or carried on the IcebergTableHandle after first resolution — so streamingSplitEstimate, planScanInternal/streamSplits, and getScanNodeProperties share one load. Alternatively wrap the connector catalog in iceberg's CachingCatalog driven by the existing meta.cache.iceberg.table.ttl-second knob, restoring the legacy IcebergExternalMetaCache table-object semantics its javadoc claims to restore (invalidation hooks already exist at IcebergConnector:524/540/552).", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981, + "note": "resolveTable: no memoization; every call -> catalogOpsResolver.apply(session).loadTable (lines 1985/1989)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "CatalogBackedIcebergCatalogOps.loadTable = raw catalog.loadTable(toTableIdentifier(...)), thin delegation, no cache" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 325, + "note": "createScanRangeLocations: getFileFormatType() -> props compute = load 1; isBatchMode() at 376 = load 2; getSplits at 422 = load 3 (batch path startSplit->streamSplits substitutes load 3)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1414, + "note": "computeBatchMode calls scanProvider.streamingSplitEstimate whenever hasSlots (memoized via isBatchModeCache at 1385-1390, so once)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 410, + "note": "streamingSplitEstimate: resolveTable AFTER the sessionBool gate at 407 (enable_external_table_batch_mode default true) — load fires on every default-config query" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 562, + "note": "planScanInternal: second independent resolveTable (sync split path); streamSplits at 452 is the streaming substitute, not additive" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1300, + "note": "getScanNodeProperties: third independent resolveTable; provider does NOT override getScanNodePropertiesResult (SPI default at ConnectorScanPlanProvider.java:455-462 wraps it once)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1776, + "note": "getOrLoadPropertiesResult memoized (cachedPropertiesResult) — per-split getFileFormatType calls hit the memo; refutes any per-split amplification but not the 3x cross-phase redundancy" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 796, + "note": "convertPredicate invalidates the props memo, but runs BEFORE the first props compute (FileQueryScanNode doFinalize order 252-253) and iceberg has no applyFilter override -> the claimed WHERE-clause double props compute (k=4) did NOT verify" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java", + "line": 57, + "note": "CachedSnapshot = {snapshotId, schemaId} only — the 'table cache' TTL knob caches a 2-long projection, not the Table object; javadoc (33) says it restores legacy IcebergExternalMetaCache table-cache semantics" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1540, + "note": "beginQuerySnapshot is the SOLE consumer of latestSnapshotCache; its loader loadTable (1541) is the +1 on TTL miss" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 692, + "note": "createCatalog builds via CatalogUtil.buildIcebergCatalog / REST session catalog with NO CachingCatalog wrap (grep: zero hits in both connector modules; iceberg-core 1.6.1 CatalogUtil.class strings contain no CachingCatalog); latestSnapshotCache wired at 188-189" + } + ] + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The core claim verifies: one iceberg SELECT on the plugin-driven path performs 3 independent full remote table loads during a single planning pass, and no layer caches the loaded Table object. (1) FileQueryScanNode.createScanRangeLocations:376 isBatchMode -> PluginDrivenScanNode.computeBatchMode:1414 streamingSplitEstimate -> IcebergScanPlanProvider:410 resolveTable; (2) split planning: non-batch FileQueryScanNode:422 getSplits -> PluginDrivenScanNode:1242 planScan -> planScanInternal:562 resolveTable, or batch startSplit -> PluginDrivenScanNode:1634 streamSplits -> IcebergScanPlanProvider:452 resolveTable (these two are ALTERNATIVES, not additive); (3) FileQueryScanNode:325 getFileFormatType -> PluginDrivenScanNode.getOrLoadPropertiesResult:1801 (memoized once per node) -> SPI default ConnectorScanPlanProvider:455-462 -> IcebergScanPlanProvider.getScanNodeProperties:1300 resolveTable. Every resolveTable (IcebergScanPlanProvider:1981-1993, no memo field) goes to IcebergCatalogOps:340-341 catalog.loadTable() on a RAW catalog — IcebergConnector.createCatalog:692-757 builds via CatalogUtil.buildIcebergCatalog/RESTSessionCatalog/S3TablesCatalog with zero CachingCatalog references in either connector module (grep-verified). Each call is a full remote metadata load (HMS getTable RPC + metadata.json fetch/parse, or REST GET). Mitigations found are all partial or off-by-default: (a) IcebergLatestSnapshotCache (wired IcebergConnector:188-189, TTL knob meta.cache.iceberg.table.ttl-second default 86400s at :129-130) caches ONLY a (snapshotId, schemaId) long pair (IcebergLatestSnapshotCache:57-65) consumed solely by IcebergConnectorMetadata.beginQuerySnapshot:1540-1545 — it amortizes that fourth load across queries but leaves the 3 planning loads untouched, despite its javadoc (:32-36) claiming to restore the legacy table-cache semantics; (b) fe-core memos (isBatchModeCache:1385-1390, cachedPropertiesResult:1776-1810) kill per-split and intra-phase repetition (the canonical #64134 per-split disaster IS fixed) but not cross-phase repetition; (c) IcebergManifestCache is consumed only when meta.cache.iceberg.manifest.enable is set, default false (IcebergConnector:159-160); (d) SDK io.manifest.cache-enabled derives true only from that same default-false knob (IcebergCatalogFactory:68,119-133) and would not cover metadata.json/HMS RPC anyway; (e) fe-connector-metastore-iceberg holds only properties/provider parsing classes, no cache; (f) IcebergConnector.getScanPlanProvider:583-592 builds a FRESH provider per resolveScanProvider() call, so no provider-instance memo exists or could currently survive. Gate check: streamingSplitEstimate's load runs whenever the scan has output slots and enable_external_table_batch_mode=true — the DEFAULT (SessionVariable.java:3041) — and IcebergScanPlanProvider:407 only skips for system tables/disabled batch mode; so k=3 holds for essentially every normal SELECT. One candidate sub-claim did NOT verify: the \"+1 with WHERE (k=4)\" — convertPredicate runs at FileQueryScanNode:252 BEFORE the first props load (:253 -> :325), so its cache invalidation (PluginDrivenScanNode:795-798) has nothing to invalidate in the standard flow; I found no props consumer executing before convertPredicate. Also \"+1 more on the streaming path\" is an overcount: streamSplits replaces planScan under batch mode, so both paths are k=3.", + "corrected_multiplicity": "k=3 per query in the default configuration on BOTH the sync path (streamingSplitEstimate + planScan + getScanNodeProperties) and the streaming path (streamingSplitEstimate + streamSplits + getScanNodeProperties; streamSplits replaces planScan, not additive); +1 amortized on IcebergLatestSnapshotCache miss (first query per table per 24h TTL / after REFRESH); the claimed k=4 with WHERE did not verify (convertPredicate precedes the first props load, so the invalidation at PluginDrivenScanNode:795-798 is a no-op in the standard flow); k=2 for slot-less scans or system tables (estimate skipped/short-circuited)", + "mitigation_found": "Partial only. (1) IcebergLatestSnapshotCache (default-ON, TTL 86400s) caches only the (snapshotId, schemaId) pin for beginQuerySnapshot — amortizes that one load, not the 3 planning loads; the meta.cache.iceberg.table.ttl-second knob users would expect to cache the table does not. (2) fe-core memoization (isBatchModeCache, cachedPropertiesResult/scanNodeProperties) fully prevents per-split and repeated intra-phase loads — the canonical per-split amplification is mitigated — but not the 3 cross-phase loads. (3) IcebergManifestCache and the SDK io.manifest.cache-enabled derivation are both default-OFF (meta.cache.iceberg.manifest.enable defaults false) and cache manifest content, not table loads. (4) No CachingCatalog wrap, no table cache in fe-connector-metastore-iceberg, no Table memo in the provider (which is rebuilt fresh per resolveScanProvider() call).", + "impact_estimate": "O(k-per-query) with k=3 x remote-IO (metastore RPC + metadata.json fetch + JSON parse per load; REST GET per load for REST catalogs) x ALL iceberg tables on the plugin-driven path under default settings (every SELECT with output slots; no rarely-true gate). Net effect: ~3x metastore/REST request load at cluster QPS versus the 1x a table-object cache would give, and 2 extra serial remote round-trips on the planning critical path — dominant for point/small queries on snapshot-heavy tables whose metadata.json is MBs.", + "fix_direction": "Memoize the loaded Table per planning pass keyed by (db, table, pinned snapshotId). Options in preference order: (a) hoist/pass-down — resolve the Table once per scan node and thread it through the three provider entry points (requires the provider to become per-node state or the handle to carry it); (b) a small Table-object cache on the long-lived IcebergConnector (next to latestSnapshotCache/manifestCache) keyed by identifier+snapshotId and honoring the existing meta.cache.iceberg.table.ttl-second knob, reusing the REFRESH invalidation hooks already present at IcebergConnector:524/540/552; (c) wrap the raw catalog in iceberg's CachingCatalog (simplest, but staleness must be reconciled with the beginQuerySnapshot pin semantics and REFRESH). Note the fresh-provider-per-call design (IcebergConnector:583-592) means a memo inside IcebergScanPlanProvider alone is insufficient.", + "evidence": [ + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1981, + "note": "resolveTable: fresh ops.loadTable per call, no memo field in the class (only local 'Table table' variables at each entry point)" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "loadTable -> catalog.loadTable(toTableIdentifier(...)) direct, no cache at the ops seam" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 410, + "note": "load #1: streamingSplitEstimate -> resolveTable (gates at :407 are only isSystemTable / enable_external_table_batch_mode, default true)" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 562, + "note": "load #2 (sync path): planScanInternal -> resolveTable; streaming alternative is streamSplits -> resolveTable at :452 (replaces, not adds)" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1300, + "note": "load #3: getScanNodeProperties -> resolveTable; reached once per node via SPI default getScanNodePropertiesResult (ConnectorScanPlanProvider.java:455-462)" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 376, + "note": "one-pass hot path: :325 getFileFormatType (props load) -> :376 isBatchMode (estimate load) -> :422 getSplits (planScan load); :252-253 shows convertPredicate runs BEFORE createScanRangeLocations, refuting the k=4 WHERE-clause double props load" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1414, + "note": "computeBatchMode invokes scanProvider.streamingSplitEstimate; planScan at :1242; getScanNodePropertiesResult at :1801-1803 memoized via cachedPropertiesResult (:1776-1810) — intra-phase/per-split repetition IS mitigated, cross-phase is not; invalidation at :795-798 fires in convertPredicate before any load in the standard flow" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java", + "line": 57, + "note": "CachedSnapshot = two longs (snapshotId, schemaId) only — NOT the Table object; javadoc :32-36 says it restores legacy IcebergExternalMetaCache table-cache semantics but the value is just the pin" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 188, + "note": "latestSnapshotCache wiring; TTL knob meta.cache.iceberg.table.ttl-second default 86400 (:129-130); single raw shared Catalog (:149, :681-690); createCatalog :692-757 has no CachingCatalog wrap (CatalogUtil.buildIcebergCatalog :755-756, RESTSessionCatalog :749-750, S3Tables :706-707); fresh provider per getScanPlanProvider() call (:583-592); manifest cache consumed only when meta.cache.iceberg.manifest.enable set, default off (:159-160)" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1540, + "note": "beginQuerySnapshot: sole consumer of latestSnapshotCache; loader does loadTable (:1541) only on miss — the one load that IS amortized (default TTL on)" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE=false; io.manifest.cache-enabled derived true only from that default-false FE knob (:119-133) — SDK manifest content cache off by default, and covers manifests, not metadata.json/HMS RPC" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java", + "line": 3041, + "note": "enableExternalTableBatchMode = true by default — the streamingSplitEstimate load (#1) runs on every normal SELECT" + } + ] + } + }, + { + "title": "Streaming split path bypasses the enabled IcebergManifestCache for exactly the large tables the cache targets (legacy honored the cache in batch mode)", + "variant": "C-cache-bypass", + "heavy_op": "IcebergScanPlanProvider.streamSplits -> scan.planFiles() (SDK reads manifest-list + ALL matched manifests remotely, per query)", + "multiplicity": "per-query on every table whose matched-manifest file count >= num_files_in_batch_mode (default 1024) when meta.cache.iceberg.manifest.enable=true; repeats the full manifest read set every query — the cache is neither read nor populated", + "entry_chain": "FileQueryScanNode.createScanRangeLocations:376 isBatchMode -> PluginDrivenScanNode.computeBatchMode:1412-1420 (streamingSplitEstimate >= threshold -> streamingBatch, NO manifest-cache gate) -> startSplit -> PluginDrivenScanNode.java:1634 streamSplits -> IcebergScanPlanProvider.streamSplits:449 -> scan.planFiles():463; the cache-honoring path planFileScanTask (IcebergScanPlanProvider.java:1681-1694, gate isManifestCacheEnabled:1832) is reachable only from the synchronous planScanInternal:627. Legacy contrast (verified in git 83585fd5097^ fe-core IcebergScanNode): doStartSplit:452->461 called planFileScanTask:537-541, which routed cache-enabled catalogs through planFileScanTaskWithManifestCache even in batch mode", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 463 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 404 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1414 + } + ], + "why_heavy": "A >=1024-file table typically has tens-to-hundreds of manifests; planFiles() re-reads all of them from object storage every query. The connector-owned IcebergManifestCache (no-TTL, capacity 100k, per-catalog) exists to make exactly these repeat reads memory-hits, and the operator explicitly enabled it — but the streaming decision fires first and unconditionally takes the SDK path, so cache hitrate for big tables is 0 and the VERBOSE 'manifest cache:' line reports nothing useful. streamingSplitEstimate could return -1 when isManifestCacheEnabled() to restore legacy routing. The fresh-FileIO-per-loadTable identity (finding 3) also defeats the SDK-level io.manifest content cache that would otherwise partially mask this.", + "gate": "meta.cache.iceberg.manifest.enable=true (default OFF) AND matched file count >= num_files_in_batch_mode AND enable_external_table_batch_mode (default true)", + "est_impact": "On cache-enabled catalogs, per-query full remote manifest re-read for all large tables — the cache delivers zero benefit where it matters most; planning latency stays at uncached levels (seconds on 10^3-10^4-manifest tables) despite the cache holding the data", + "confidence": "high", + "lens": "cache-audit", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Independently re-derived the full chain from the actual code. (1) Heaviness holds: IcebergScanPlanProvider.streamSplits line 463 calls scan.planFiles() unconditionally — per the SDK cost model this reads the manifest-list plus ALL matched manifests via remote FileIO every query; the method's own javadoc (lines 444-445) admits 'Bypasses the manifest cache'. (2) Multiplicity holds as claimed: per-query on every table whose matched-manifest file count >= num_files_in_batch_mode (default 1024, line 142/422). The routing is airtight: FileQueryScanNode.createScanRangeLocations:376 isBatchMode -> PluginDrivenScanNode.computeBatchMode:1414 calls connector streamingSplitEstimate; IcebergScanPlanProvider.streamingSplitEstimate:404-437 gates only on systemTable/enable_external_table_batch_mode/null-snapshot/format-v3/count-pushdown — NO isManifestCacheEnabled() gate — so estimate>=0 forces streamingBatch=true (1416-1419) -> startSplit:1498 -> startStreamingSplit:1600 -> streamSplits at 1634. The cache-honoring path planFileScanTask (1681, gate isManifestCacheEnabled 1683, cache planning 1687->1706 reading each manifest through IcebergManifestCache) is reachable only from the synchronous planScanInternal:627, which streaming preempts. (3) Legacy contrast verified in git 83585fd5097^ fe-core IcebergScanNode: isBatchMode (885-937) had no cache gate, and the batch entry doStartSplit:452 -> line 461 planFileScanTask -> 537-542 routed cache-enabled catalogs through planFileScanTaskWithManifestCache (manifest reads via IcebergManifestCacheLoader at 651/684); the lazy planFiles batch flavor (splitFiles:555-558) was reachable only with the cache off. So legacy batch mode DID honor the cache; the new code silently converts cache-on large-table planning to cache-off behavior — repeat full remote manifest reads every query, cache neither read nor populated, 0% hit rate on exactly the tables the cache targets. The new javadoc's justification ('legacy's lazy batch path only ran with the manifest cache off') is literally true of the lazy sub-path but misleading as parity evidence, since legacy reached that sub-path only because cache-on batch went through the cache first. The bypass is a deliberate OOM-protection trade (cache planning materializes the task list), which is a design tension, not a mitigation — the regression vs legacy cache effectiveness is real. This is variant C (cache bypass) of the problem class; gates noted below do not refute (rare gate still counts per audit rules).", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 463, + "note": "streamSplits: unconditional TableScanUtil.splitFiles(scan.planFiles(), sliceSize) — remote manifest-list + all matched manifests read, no cache; javadoc lines 444-445 admits 'Bypasses the manifest cache'" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 404, + "note": "streamingSplitEstimate (404-437): gates = systemTable/enable_external_table_batch_mode (407), null snapshot (413), formatVersion>=3 (416), count pushdown (419), threshold num_files_in_batch_mode default 1024 (422; constant at 142). NO isManifestCacheEnabled() gate — cache-enabled catalogs stream like everyone else" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1683, + "note": "planFileScanTask gate: isManifestCacheEnabled() routes to planFileScanTaskWithManifestCache (1687->1706, per-manifest reads via manifestCache.getManifestCacheValue at 1738); grep confirms sole call site is planScanInternal line 627 — unreachable once streaming fires" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1832, + "note": "isManifestCacheEnabled: manifestCache wired AND meta.cache.iceberg.manifest.enable (default false, constant DEFAULT_MANIFEST_CACHE_ENABLE at line 201) with non-zero ttl/capacity" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1414, + "note": "computeBatchMode: calls scanProvider.streamingSplitEstimate; estimate>=0 sets streamingBatch=true (1416-1419) with no manifest-cache consideration; startSplit:1498 -> startStreamingSplit:1600 -> streamSplits at 1634" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 376, + "note": "createScanRangeLocations: isBatchMode() branch dispatches to SplitAssignment + startSplit path, preempting the synchronous getSplits/planScan (cache-honoring) path" + }, + { + "file": "git 83585fd5097^ fe/fe-core/.../iceberg/source/IcebergScanNode.java", + "line": 461, + "note": "LEGACY: doStartSplit (batch entry, startSplit:441) called planFileScanTask(scan); planFileScanTask 537-542 routed cache-enabled catalogs to planFileScanTaskWithManifestCache (IcebergManifestCacheLoader reads at 651/684) — legacy batch mode honored the cache; lazy planFiles batch flavor (splitFiles 555-558) reachable only with cache off; isBatchMode 885-937 has no cache gate, threshold check at 929" + } + ], + "corrected_multiplicity": "per-query (as claimed): full matched-manifest remote read set repeated on every query against any table with matched file count >= num_files_in_batch_mode (default 1024) on a manifest-cache-enabled catalog; the cache is neither read nor populated on this path, so small tables (<1024 files, synchronous path) get cache hits while the large tables the cache targets get 0%", + "mitigation_found": "None on the streaming path. The bypass is deliberate (javadoc 444-446: cache planning materializes the full task list, defeating streaming's OOM protection) but that is a design trade, not a cache hit — and its legacy-parity claim is misleading since legacy cache-on batch mode went through the cache (materializing, pumped async with backpressure). No SDK-level fallback either: iceberg's io.manifest content cache is off by default and would additionally require a stable FileIO identity.", + "fix_direction": "Either (a) restore legacy routing: make streamingSplitEstimate return -1 when isManifestCacheEnabled(), so cache-enabled catalogs take planScanInternal -> planFileScanTask -> planFileScanTaskWithManifestCache (accepts legacy's materialization cost, exact parity); or (b) strictly better than legacy: teach IcebergStreamingSplitSource to enumerate per-manifest through IcebergManifestCache (manifest-granular lazy iteration keeps FE heap bounded AND populates/reads the cache). Option (a) is the low-risk parity fix; note it trades streaming OOM protection for cache hits exactly as legacy did, which the operator opted into by enabling the cache.", + "impact_estimate": "Gated on operator opt-in (meta.cache.iceberg.manifest.enable=true, default OFF) + enable_external_table_batch_mode (default true) + matched files >= 1024. Where it applies: every query on every large table re-reads tens-to-hundreds of matched manifest files from object storage (one remote GET + avro decode each) instead of memory hits — planning latency stays at uncached levels (seconds on 10^3-10^4-manifest tables) despite the enabled cache, and the VERBOSE 'manifest cache:' explain line reports zero activity for exactly the tables the cache was enabled for." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The bypass is real and unmitigated. Current code: FileQueryScanNode.createScanRangeLocations:376 -> PluginDrivenScanNode.computeBatchMode:1414-1419 (streamingSplitEstimate >= 0 -> streaming batch, no cache gate at either layer) -> startStreamingSplit:1633-1634 -> IcebergScanPlanProvider.streamSplits:449 -> raw scan.planFiles() at :463. The IcebergManifestCache is consulted only inside planFileScanTaskWithManifestCache (:1738/:1761), reachable solely from the synchronous planScanInternal:627 -> planFileScanTask:1681-1694 behind isManifestCacheEnabled (:1832-1841) — a path taken only by tables BELOW the num_files_in_batch_mode threshold (default 1024). So on a cache-enabled catalog the cache serves only small tables and delivers zero benefit (neither read nor populated) for the large tables it was built for; the VERBOSE 'manifest cache:' stats show nothing for them. The streamSplits Javadoc (:440-446) admits the bypass but its parity claim ('legacy's lazy batch path only ran with the manifest cache off') is misleading: verified in git 83585fd5097^ fe-core IcebergScanNode, legacy isBatchMode (885-937) had no cache gate, and doStartSplit:452 -> planFileScanTask:461 -> :537-542 routed cache-enabled catalogs through planFileScanTaskWithManifestCache:595 even in batch mode (materializing, cache-honoring); only the cache-OFF batch branch was lazy (splitFiles :555-558). The migration inverted the priority (laziness over cache) for cache-enabled large tables — a genuine behavior regression versus legacy. Mitigation hunt found two candidate layers, both ineffective: (1) the Doris-level IcebergManifestCache — bypassed as above; (2) the SDK-level manifest content cache — IcebergCatalogFactory.appendManifestCacheProperties:119-133 does derive io.manifest.cache-enabled=true when the Doris cache is on, but that cache is weak-keyed per FileIO INSTANCE, and IcebergCatalogOps.loadTable:340-341 is a raw catalog.loadTable() creating fresh TableOperations+FileIO per call (CatalogUtil.buildIcebergCatalog at IcebergConnector.java:756 does not CachingCatalog-wrap; zero CachingCatalog/table-cache hits in fe-connector-iceberg + fe-connector-metastore-iceberg), so the content cache is reborn every query and never hits across queries. The finding's gate is off-by-default (meta.cache.iceberg.manifest.enable defaults false, IcebergCatalogFactory.java:68), which I note per the audit rules — but the affected population is exactly the operators who explicitly enabled the cache.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 463, + "note": "streamSplits builds tasks via TableScanUtil.splitFiles(scan.planFiles(), sliceSize) — raw SDK remote read of manifest-list + all matched manifests; manifestCache is never touched (its only read sites are :1738 and :1761 inside planFileScanTaskWithManifestCache)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 444, + "note": "Javadoc admits 'Bypasses the manifest cache' and claims legacy parity ('legacy's lazy batch path only ran with the manifest cache off') — true of the lazy branch only; conceals that legacy cache-enabled batch queries went through the cache" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 404, + "note": "streamingSplitEstimate (404-438): gates are system-table, enable_external_table_batch_mode, snapshot!=null, formatVersion<3, servable count pushdown, fileCount>=num_files_in_batch_mode — NO isManifestCacheEnabled gate" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1683, + "note": "planFileScanTask: if (!isManifestCacheEnabled()) splitFiles else planFileScanTaskWithManifestCache — the cache-honoring gate, reachable only from the synchronous planScanInternal (:627), i.e., only tables below the streaming threshold" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1414, + "note": "computeBatchMode calls scanProvider.streamingSplitEstimate; estimate>=0 -> streamingBatch=true -> return true; no cache-related gate (correctly connector-agnostic — the missing gate belongs in the connector estimate)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1634, + "note": "startStreamingSplit -> scanProvider.streamSplits(...) on the async schedule executor — the only caller of the bypassing path" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java", + "line": 376, + "note": "createScanRangeLocations: if (isBatchMode()) takes the split-assignment/startSplit path — hot-path entry" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 131, + "note": "Mitigation candidate #2: derives SDK io.manifest.cache-enabled=true when meta.cache.iceberg.manifest.enable is on — but ineffective across queries (SDK content cache is weak-keyed per FileIO instance)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "loadTable = raw catalog.loadTable(): fresh TableOperations+FileIO per call; no CachingCatalog wrap anywhere (IcebergConnector.java:756 uses CatalogUtil.buildIcebergCatalog; grep CachingCatalog = 0 hits in fe-connector-iceberg + fe-connector-metastore-iceberg), so the derived SDK content cache never persists" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/source/IcebergScanNode.java", + "line": 537, + "note": "LEGACY at git 83585fd5097^ (file deleted at HEAD; read from git show): planFileScanTask routes cache-enabled -> planFileScanTaskWithManifestCache even from doStartSplit (:452->:461); isBatchMode (:885-937) has no cache gate; lazy splitFiles batch branch (:555-558) ran only cache-off — legacy honored the cache in batch mode, candidate's legacy claim verified correct" + } + ], + "corrected_multiplicity": "per-query x O(matched manifests) remote reads (tens to hundreds of manifest files + manifest-list per query, repeated every query with 0% cache hit), on every table whose matched file count >= num_files_in_batch_mode (default 1024) in a catalog with meta.cache.iceberg.manifest.enable=true; small tables below the threshold still get the cache. Minor rider: the streaming path loads the table twice per query (resolveTable in streamingSplitEstimate:410 and again in streamSplits:452 — two metastore RPC + metadata.json reads).", + "mitigation_found": "Two candidate mitigations located, neither effective: (1) the connector-owned IcebergManifestCache (no-TTL, capacity 100k, per-catalog) — wired into the provider but consulted only on the synchronous planScanInternal path; streamSplits neither reads nor populates it. (2) IcebergCatalogFactory.appendManifestCacheProperties (119-133) derives SDK-level io.manifest.cache-enabled=true from the same catalog property — but the SDK ManifestFiles content cache is weak-keyed per FileIO instance and IcebergCatalogOps.loadTable creates a fresh FileIO every call (no CachingCatalog wrap, no connector/metastore table cache), so it is reborn empty each query. Operator workarounds exist (enable_external_table_batch_mode=false, or raising num_files_in_batch_mode) but sacrifice streaming's OOM protection and are not defaults. Gate on the whole finding: meta.cache.iceberg.manifest.enable defaults to FALSE, so only catalogs where the operator explicitly enabled the manifest cache are affected — exactly the deployments that asked for cached planning.", + "impact_estimate": "O(matched-manifests) remote-IO per query; trigger breadth: only manifest-cache-enabled catalogs (opt-in, default off) x only tables with >=1024 matched files x format-version<3 x batch mode on (session default on). Where triggered, planning stays at cold-cache latency every query (full remote manifest re-read, typically seconds on 10^3-10^4-manifest tables) despite the operator-enabled cache holding — or being able to hold — the data; cache hit rate for these tables is exactly 0 and the VERBOSE 'manifest cache:' profile line is empty/zero for them. Legacy served the same reads from memory after first query (at the cost of materializing the task list).", + "fix_direction": "Two options, in increasing quality: (a) one-line legacy-parity routing fix — streamingSplitEstimate returns -1 when isManifestCacheEnabled(), forcing cache-enabled catalogs onto the synchronous materializing cache path exactly as legacy did (restores cache hits, loses streaming OOM protection for those catalogs, matching legacy behavior); (b) better — make the streaming source cache-aware: iterate matching data manifests and read each through IcebergManifestCache.getManifestCacheValue one manifest at a time (per-manifest laziness bounds memory at O(one manifest's entries) instead of O(all tasks)), applying the same ManifestEvaluator/InclusiveMetricsEvaluator/ResidualEvaluator pruning as planFileScanTaskWithManifestCache — keeps both the cache and the OOM protection. Either way, also fix the misleading streamSplits Javadoc parity claim." + } + }, + { + "title": "COUNT(*) pushdown path bypasses the enabled IcebergManifestCache: planCountPushdown calls scan.planFiles() directly", + "variant": "C-cache-bypass", + "heavy_op": "planCountPushdown -> scan.planFiles() (SDK manifest-list + matched-manifest remote reads; ParallelIterable eagerly submits all manifest reads although only the first task is kept)", + "multiplicity": "per-query for every servable COUNT(*) pushdown query when the manifest cache is enabled (planScanInternal:594-599 returns before reaching the cache-gated planFileScanTask at :627)", + "entry_chain": "PluginDrivenScanNode.getSplits:1242 planScan(countPushdown=true) -> IcebergScanPlanProvider.planScanInternal:594-598 (getCountFromSnapshot >= 0) -> planCountPushdown:1021 -> scan.planFiles():1024; the manifest-cache route planFileScanTask:1681 (isManifestCacheEnabled:1832 -> planFileScanTaskWithManifestCache:1706 reading via manifestCache.getManifestCacheValue:1738/1761) is only reached at :627, after the count branch has already returned. Legacy contrast: legacy IcebergScanNode's count path byte-split via planFileScanTask (git 83585fd5097^ line 853), which honored the cache", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1024 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 594 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java", + "line": 118 + } + ], + "why_heavy": "The count itself is answered from the in-memory snapshot summary (cheap), but emitting the single collapsed range opens a fresh SDK planFiles whose iterator construction fans out remote reads of all matched manifests — none served from the already-populated IcebergManifestCache. On a cache-warm catalog, a COUNT(*) on a large table pays near-full uncached planning IO for one range.", + "gate": "meta.cache.iceberg.manifest.enable=true (default OFF) AND count pushdown servable (no equality deletes; position deletes only with ignore_iceberg_dangling_delete)", + "est_impact": "Per COUNT(*) query on cache-enabled catalogs: remote read of all matched manifests instead of memory hits — turns a metadata-only query's planning from ~ms into the table's full manifest-scan latency", + "confidence": "high", + "lens": "cache-audit", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Bypass verified by direct read: planScanInternal's count branch (IcebergScanPlanProvider.java:594-599) returns via planCountPushdown before reaching the cache-gated planFileScanTask at :627; planCountPushdown opens a raw scan.planFiles() (:1024) and keeps only the first FileScanTask. The Doris IcebergManifestCache route (planFileScanTask :1681-1694 -> planFileScanTaskWithManifestCache :1706, manifestCache.getManifestCacheValue :1738/:1761) is structurally unreachable for a servable COUNT(*). streamingSplitEstimate :419-421 returns -1 for servable counts, so such queries always take this synchronous path; fe-core drives it once per scan node (PluginDrivenScanNode.java:1241-1244). Legacy regression confirmed in git 83585fd5097^: legacy IcebergScanNode's count branch (line 853) enumerated tasks via planFileScanTask (cache gate at 537-542), i.e. it honored the cache. getCountFromSnapshot itself is in-memory (snapshot.summary(), :1913-1918) — the planFiles call exists only to fetch one file path for the collapsed range, so the heavy op does far more work than the information needed. Heaviness: scan.planFiles() = manifest-list read + ALL matched delete-manifest reads (DeleteFileIndex builds before the first data task) + O(worker-pool) data-manifest reads before close — remote IO per the SDK cost model. One nuance refines but does not refute: the same FE gate also derives the SDK's byte-level cache io.manifest.cache-enabled=true (IcebergCatalogFactory.java:119-133), and iceberg-core 1.10.1 bytecode confirms ManifestGroup->ManifestFiles.read honors it — BUT for the default HMS flavor (connector never sets io-impl; grep verified) HiveCatalog.initialize creates new HadoopFileIO(conf) WITHOUT properties (verified via javap), so io.properties() is empty, ManifestFiles.cachingEnabled=false, and no SDK cache applies: the count path pays real remote manifest IO per query while the enabled, warm Doris cache is skipped. REST catalogs returning per-table config/vended credentials build a fresh FileIO per table load (RESTSessionCatalog.tableFileIO bytecode), leaving the weak-keyed SDK ContentCache cold across queries too. HADOOP/GLUE/JDBC/plain-REST flavors get the SDK byte cache (FileIO initialized with properties), reducing the bypass cost to per-query avro decode CPU — still redundant work vs the parsed-object cache, but not remote IO. Both heaviness and claimed multiplicity (per-query on every servable COUNT(*) with the gate on) hold; the finding's \"eagerly submits all manifest reads\" is overstated for the bounded 1.10.1 ParallelIterable, and impact should be flavor-qualified.", + "evidence": [ + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 594, + "note": "count branch: countPushdown && getCountFromSnapshot>=0 returns planCountPushdown at :597 BEFORE the cache-gated planFileScanTask at :627" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1024, + "note": "planCountPushdown opens raw scan.planFiles() (try-with-resources), consumes only the FIRST FileScanTask (:1025-1029) — never routed through planFileScanTask/IcebergManifestCache" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1683, + "note": "planFileScanTask: isManifestCacheEnabled() gate -> planFileScanTaskWithManifestCache (:1687); cache reads at :1738 (delete manifests) and :1761 (data manifests) via manifestCache.getManifestCacheValue — the route the count path skips" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 419, + "note": "streamingSplitEstimate returns -1 for a servable COUNT(*) pushdown, so these queries always take the synchronous planScan path containing the bypass" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 201, + "note": "gate: meta.cache.iceberg.manifest.enable default false (DEFAULT_MANIFEST_CACHE_ENABLE); enable formula at isManifestCacheEnabled :1832-1841 — rare-gate noted, does not refute" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1241, + "note": "entry: countPushdown = TPushAggOp.COUNT && !applySample -> planScan(..., countPushdown) at :1242-1244, once per query per scan node; connector overload :380-388 -> planScanInternal(countPushdown)" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java", + "line": 129, + "note": "getManifestCacheValue(manifest, table, queryId): parsed DataFile/DeleteFile lists cached in MetaCacheEntry (no TTL, cap 100000) — the populated cache the count path never consults" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 125, + "note": "MITIGATION missed by the finding: same gate derives io.manifest.cache-enabled=true into SDK catalog props for all flavors (:119-133 via buildBaseCatalogProperties :108). Verified in iceberg-core 1.10.1 bytecode: ManifestFiles.cachingEnabled reads this from io.properties() and ManifestGroup reads manifests via ManifestFiles.read (cache-aware). HOWEVER HiveCatalog.initialize (iceberg-hive-metastore 1.10.1, javap) builds new HadoopFileIO(conf) WITHOUT properties when io-impl unset (connector never sets io-impl — grep of connector source empty), so the HMS flavor gets NO SDK cache -> real remote manifest IO per COUNT query. RESTSessionCatalog.tableFileIO returns a fresh per-load FileIO when the server sends table config/credentials -> weak-keyed SDK ContentCache cold across queries. HADOOP/GLUE/JDBC use CatalogUtil.loadFileIO(properties) -> SDK byte cache active -> bypass cost drops to per-query avro-decode CPU" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1913, + "note": "getCountFromSnapshot reads scan.snapshot().summary() — in-memory; the count itself needs zero manifest IO, so ALL manifest reads in planCountPushdown serve only to name one data file for the collapsed range. Legacy contrast verified: git 83585fd5097^ IcebergScanNode:853 wrapped the count branch in planFileScanTask(scan), cache-gated at legacy :537-542" + } + ], + "corrected_multiplicity": "per-query (k=1 per servable COUNT(*) pushdown query, on every such query while meta.cache.iceberg.manifest.enable=true) — as claimed; problem-class variant C cache-bypass, exempt from the inherent-planFiles carve-out because the enabled cache (and legacy code) served exactly this read from memory and only one file path is actually needed", + "fix_direction": "Restore legacy parity: in planCountPushdown (or before calling it), enumerate via the existing cache-gated planFileScanTask(scan, session, table, filter) and take the first task, instead of raw scan.planFiles() — one-line-scale change reusing the :1681 gate with its built-in SDK-scan fallback. Cheaper still: with the cache enabled, build the single range from the first data file of the first cached data manifest (no full DeleteFileIndex needed since the count is already netted from the summary).", + "impact_estimate": "Flavor-qualified from the candidate's estimate: on HMS-flavor catalogs (Doris's most common iceberg deployment; no io-impl => no SDK byte cache) and on REST catalogs with per-table config/vended credentials, every servable COUNT(*) pays remote manifest-list + all matched delete manifests + O(worker-pool) data-manifest reads despite a warm enabled cache — planning goes from ~ms (memory hits) to the table's manifest-fetch latency (seconds on high-latency object stores with hundreds of manifests), scaling with COUNT(*) QPS. On hadoop/glue/jdbc/plain-REST flavors the SDK byte-level cache absorbs the remote IO and the residual waste is per-query avro decode of the touched manifests (CPU-only, moderate). \"Near-full uncached planning IO\" is slightly overstated: 1.10.1's bounded ParallelIterable + first-task early close cancels the tail of data-manifest reads.", + "mitigation_found": "Partial, flavor-dependent: IcebergCatalogFactory.appendManifestCacheProperties (:119-133) derives the iceberg SDK's io.manifest.cache-enabled=true from the same gate, and scan.planFiles() reads manifests through the cache-aware ManifestFiles.read — effective (byte-level, decode still repeated) for hadoop/glue/jdbc and plain-REST flavors whose FileIO is initialized with catalog properties and shared. NOT effective for the default HMS flavor (HiveCatalog builds HadoopFileIO(conf) without properties; connector never sets io-impl) nor for REST catalogs issuing per-table FileIO (fresh weak-keyed ContentCache per table load). No other memoization on the count path." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The bypass is real and verified line-by-line: the COUNT(*) branch (planScanInternal:594-599) returns through planCountPushdown before reaching the cache-gated planFileScanTask at :627, and planCountPushdown:1024 opens scan.planFiles() directly — the FE IcebergManifestCache (parsed manifests, no-TTL, cap 100k) is never consulted. Legacy parity contrast verified against git 83585fd5097^: legacy IcebergScanNode's count path (line 853) routed through planFileScanTask (537→542) which honored the manifest cache, so this is a regression on cache-enabled catalogs. The mitigation hunt found ONE mechanism the finder missed: IcebergCatalogFactory.appendManifestCacheProperties (119-133) derives the SDK-level io.manifest.cache-enabled=true from the SAME FE gate, and the SDK plan path does read manifests via ManifestFiles.read (verified in ManifestGroup$1 bytecode), so the SDK ContentCache can absorb the remote IO — but it is only PARTIAL: (a) it is inactive on the HMS flavor (and hadoop/jdbc) because iceberg 1.10.1 HiveCatalog.initialize constructs `new HadoopFileIO(conf)` without catalog properties when io-impl is unset (verified by decompiling the 1.10.1 jar; no io-impl/FILE_IO_IMPL is set anywhere in fe-connector-iceberg or fe-connector-metastore-iceberg), so cachingEnabled(io)=false and the count path pays full remote manifest IO; (b) where active (REST — RESTSessionCatalog.initialize gets full props at IcebergConnector:813 — and Glue/S3FileIO) it caches raw bytes with a 60s default expiration (IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT = 60_000, verified from clinit bytecode), 100MB total / 8MB per-manifest caps, and repays avro decode CPU every query, vs the bypassed FE cache's parsed no-TTL entries. Cost shape confirmed: getCountFromSnapshot (1913-1918) is summary-only (cheap); the heavy part is solely the one-range emission, where DeleteFileIndex$Builder eagerly reads ALL delete manifests via Tasks.foreach+executor before the first data task can be emitted (relevant under ignore_iceberg_dangling_delete with position deletes), and ParallelIterable submits all matched data-manifest reads to the worker pool, with close-after-first-task cancelling only unstarted reads. Entry chain verified: PluginDrivenScanNode.getSplits:1241-1244 computes countPushdown and calls planScan exactly once; streamingSplitEstimate:419-421 exits from the summary for servable counts (no extra heavy op). Gate honestly noted: DEFAULT_MANIFEST_CACHE_ENABLE=false (:201), so the default configuration has no cache to bypass and the single planFiles is inherent planning cost — the finding only bites on catalogs that explicitly enabled the manifest cache, exactly the population that opted in to fast metadata planning. Verdict CONFIRMED because the primary mechanism (FE manifest cache) is fully bypassed and the secondary mechanism (SDK ContentCache) is flavor-dependent (dead on the common HMS-on-HDFS deployment), short-lived (60s), and byte-level only.", + "evidence": [ + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 594, + "note": "countPushdown branch: getCountFromSnapshot>=0 returns via planCountPushdown at :597 BEFORE the cache-gated planFileScanTask at :627 — bypass confirmed" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1024, + "note": "planCountPushdown opens scan.planFiles() directly (SDK path) and keeps only the first FileScanTask (1025-1030); no manifestCache consultation anywhere in the method" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1683, + "note": "planFileScanTask is the cache gate (isManifestCacheEnabled -> planFileScanTaskWithManifestCache:1687, which reads every manifest through manifestCache.getManifestCacheValue at 1738/1761) — the route the count branch skips" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 201, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE=false — the whole finding is gated on meta.cache.iceberg.manifest.enable=true (off by default); ttl/capacity defaults 48h/1024 feed only the enable formula" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java", + "line": 129, + "note": "the bypassed mechanism: per-catalog cache of PARSED DataFile/DeleteFile lists, no-TTL, capacity 100k (lines 52-65, 104-111) — warm entries would make the count path pure memory" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 125, + "note": "MITIGATION FOUND (partial): the same FE gate derives SDK io.manifest.cache-enabled=true (:131) into catalog properties when the user did not set it — enables iceberg's ManifestFiles ContentCache for FileIOs initialized from catalog props" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 756, + "note": "catalogProps (with derived SDK flag) feed CatalogUtil.buildIcebergCatalog; REST flavor passes full props via sessionCatalog.initialize at :813, so REST FileIO carries the flag (SDK cache active); HMS does not" + }, + { + "file": "/tmp/claude-1000/-mnt-disk1-yy-git-wt-catalog-spi/7c1fb5c4-fc44-4add-b72e-ea8606d9be3a/scratchpad/iceb/HiveCatalog.txt", + "line": 148, + "note": "iceberg 1.10.1 HiveCatalog.initialize bytecode: io-impl==null -> `new HadoopFileIO(conf)` (properties NOT passed, :151-159) vs loadFileIO with props (:171) — SDK ContentCache inactive for the common HMS flavor; grep confirms no io-impl/FILE_IO_IMPL set anywhere in fe-connector-iceberg or fe-connector-metastore-iceberg. Also verified from iceberg-core 1.10.1: IO_MANIFEST_CACHE_EXPIRATION_INTERVAL_MS_DEFAULT=SECONDS.toMillis(60), MAX_TOTAL_BYTES=100MB, MAX_CONTENT_LENGTH=8MB; ManifestGroup$1 reads via ManifestFiles.read (cache applies when enabled); DeleteFileIndex$Builder eagerly reads ALL delete manifests via Tasks.foreach+executeWith before the first task" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java", + "line": 1242, + "note": "entry chain verified: getSplits computes countPushdown (=COUNT agg && !applySample, :1241) and calls planScan exactly once per query; streamingSplitEstimate exits at IcebergScanPlanProvider:419-421 from the summary for servable counts (no second heavy op)" + }, + { + "file": "/mnt/disk1/yy/git/wt-catalog-spi/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 1913, + "note": "getCountFromSnapshot reads only scan.snapshot().summary() (in-memory) — the count itself is cheap; the heavy op is exclusively the single-range-emission planFiles. Legacy contrast verified via `git show 83585fd5097^`: old IcebergScanNode count path (line 853) -> planFileScanTask (537) -> planFileScanTaskWithManifestCache (542) honored the FE cache" + } + ], + "corrected_multiplicity": "per-query-constant (k=1 SDK planFiles per servable COUNT(*) query) — variant C cache-bypass, not loop amplification: the delta is remote-IO-vs-memory on manifest-cache-enabled catalogs, paid once per COUNT(*) statement", + "mitigation_found": "Partial. (1) The FE IcebergManifestCache — the mechanism the finding is about — is fully bypassed; confirmed. (2) Missed by the finder: IcebergCatalogFactory.appendManifestCacheProperties (IcebergCatalogFactory.java:119-133) derives the SDK-level io.manifest.cache-enabled=true from the same meta.cache.iceberg.manifest.* gate, and the SDK plan path reads manifests through ManifestFiles.read, so iceberg's ContentCache absorbs the remote reads — but only for flavors whose FileIO is initialized with catalog properties (REST via RESTSessionCatalog.initialize, Glue/S3FileIO). For the HMS flavor with no io-impl (the common HMS-on-HDFS deployment) HiveCatalog 1.10.1 builds `new HadoopFileIO(conf)` without properties, so the SDK cache is dead and the bypass costs full remote manifest IO. Even where active, the SDK cache holds raw bytes for 60s (default expiration) with 8MB-per-manifest/100MB-total caps and repays avro decode per query — strictly weaker than the bypassed no-TTL parsed-entry FE cache. No other layer (IcebergLatestSnapshotCache, metastore table caches) covers manifest reads.", + "impact_estimate": "O(1) heavy op per query x remote-IO (HMS flavor; degraded-to-local-CPU on REST/Glue within the SDK cache's 60s window) x narrow trigger breadth: only catalogs with meta.cache.iceberg.manifest.enable=true (default OFF), only full-table COUNT(*) with pushdown servable (no equality deletes; position deletes only under ignore_iceberg_dangling_delete). Per such query on an HMS-flavor cache-enabled catalog: manifest-list + ALL delete manifests (eager DeleteFileIndex, when position deletes exist) + a worker-pool fan-out of data-manifest reads (all submitted, unstarted ones cancelled on close after the first task) — i.e., a metadata-answered query pays up to a near-full uncached manifest scan instead of memory hits; planning goes from ~ms to the table's manifest-scan latency (seconds on large tables / slow object storage). Not a correctness issue and not N-times amplification — a per-query cache-bypass regression vs legacy.", + "fix_direction": "pass-down/route fix (legacy parity): source the count-collapse's single range through the cache-gated planFileScanTask instead of a raw scan.planFiles() — i.e., in the countPushdown branch call planFileScanTask(scan, session, table, filter) and take the first task (exactly what legacy IcebergScanNode:853 did), so an enabled IcebergManifestCache serves the manifests from memory. Cheaper variant: when isManifestCacheEnabled(), short-circuit to the first live DataFile of the first matching data manifest via manifestCache.getManifestCacheValue (the collapsed range only needs one file; no DeleteFileIndex needed since the range carries table_level_row_count). Keep the SDK planFiles path as-is for the cache-disabled default." + } + }, + { + "title": "rewrite_data_files: one whole-table planFiles() re-scan PER rewrite group in registerRewriteSourceFiles", + "variant": "A-loop-amplification", + "heavy_op": "table.newScan().useSnapshot(startingSnapshotId).planFiles() — full manifest-list + all-manifests remote read at the pinned OCC snapshot (IcebergConnectorTransaction.java:379-383)", + "multiplicity": "k-times-per-query where k = number of bin-packed rewrite groups: ConnectorRewriteDriver.run() STEP 3 loops `for (ConnectorRewriteGroup group : groups)` (ConnectorRewriteDriver.java:143) and calls connectorTx.registerRewriteSourceFiles(group.getDataFilePaths()) once per group (line 144); every call performs its own whole-table scan. For a large-table compaction k is easily 10s-100s of groups.", + "entry_chain": "ConnectorExecuteAction -> ConnectorRewriteDriver.run() (fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java:143-144) -> IcebergConnectorTransaction.registerRewriteSourceFiles (fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java:362) -> context.executeAuthenticated -> table.newScan().planFiles() (IcebergConnectorTransaction.java:379-383)", + "files": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java", + "line": 143 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java", + "line": 383 + } + ], + "why_heavy": "planFiles() reads the manifest list plus every matching manifest through FileIO (remote IO, O(table metadata)). The scan result is invariant across the loop: every iteration scans the SAME pinned snapshot, only the wanted-path set differs. It is also chain-redundant: RewriteDataFilePlanner.planAndOrganizeTasks already ran planFiles() once at the same snapshot (rewrite/RewriteDataFilePlanner.java:141) and held the exact DataFile objects; the neutral SPI reduced them to String paths (IcebergProcedureOps.toConnectorRewriteGroup, IcebergProcedureOps.java:227-233), forcing the connector to re-derive them per group.", + "est_impact": "One ALTER TABLE ... EXECUTE rewrite_data_files performs G+1 whole-table manifest scans instead of 1 (plus the getFileFormat scans of finding 2). On a table with thousands of manifests and G~50-200 groups this turns a seconds-long registration step into minutes of serial remote IO before commit. Fix is a trivial hoist: union all groups' paths into ONE registerRewriteSourceFiles call (the mismatch check already works on totals), or key the matched DataFiles from the single planning scan.", + "gate": "Only ALTER TABLE ... EXECUTE rewrite_data_files (post-P6.6-flip path); ungated by table properties — happens on every distributed rewrite with >1 group.", + "confidence": "high", + "lens": "write-commit-path", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Both heaviness and multiplicity hold exactly as claimed. Heaviness: registerRewriteSourceFiles performs an UNFILTERED table.newScan().useSnapshot(startingSnapshotId).planFiles() per call (IcebergConnectorTransaction.java:379-383) — manifest-list + all manifests via FileIO, remote IO per the SDK cost model, and the code's own comment (lines 376-377) says so; iceberg's manifest cache is off by default so nothing mitigates it. Multiplicity: ConnectorRewriteDriver.run() STEP 3 calls it once per bin-packed group inside `for (ConnectorRewriteGroup group : groups)` (ConnectorRewriteDriver.java:143-144), and grep confirms this loop is the sole production caller; the driver is live via ConnectorExecuteAction.java:161. Loop-invariance: startingSnapshotId is pinned once at REWRITE begin (line 267) and never changes across iterations, so all k scans read the identical snapshot — only the wanted-path filter differs; there is no cross-call memoization (matched/wanted are per-call locals). The finding is a textbook variant-A loop amplification. One nuance downgrades only the secondary chain-redundancy claim: the planner's earlier planFiles (RewriteDataFilePlanner.java:141) ran at the plan-time current snapshot, while the connector re-scans the begin-time OCC anchor — these can differ under concurrent commits, so ONE re-derive scan has genuine OCC-validation value and is not strictly redundant with planning. That does not refute the finding: k−1 of the k scans are pure waste, and the per-group repetition (not the single re-derive) is the reported defect.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java", + "line": 143, + "note": "STEP 3: `for (ConnectorRewriteGroup group : groups)` then line 144 `connectorTx.registerRewriteSourceFiles(group.getDataFilePaths())` — one call per bin-packed group; grep shows this is the ONLY production caller of registerRewriteSourceFiles." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java", + "line": 383, + "note": "Each call runs `scan.planFiles()` on `table.newScan()` (line 379) pinned via `useSnapshot(startingSnapshotId)` (line 381), iterating EVERY FileScanTask of the table; comment at 376-377 confirms 'reads the manifest list + manifests (remote IO)'. `wanted`/`matched` (lines 373-374) are per-call locals — no memoization across calls." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java", + "line": 267, + "note": "applyBeginGuards for WriteOperation.REWRITE pins startingSnapshotId ONCE at begin; it is invariant across the driver's registration loop, so all k scans read the identical snapshot — the scan result is loop-invariant." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/rewrite/RewriteDataFilePlanner.java", + "line": 141, + "note": "Planning already ran `tableScan.planFiles()` once holding the exact DataFile objects; nuance: at the plan-time CURRENT snapshot (lines 113-114), which may differ from the begin-time OCC anchor, so one connector re-scan has validation value — but only one." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergProcedureOps.java", + "line": 227, + "note": "toConnectorRewriteGroup reduces each group's DataFiles to raw String paths (lines 228-230) crossing the neutral SPI wall — the reason the connector must re-derive DataFile objects at all." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java", + "line": 161, + "note": "Live production entry: ConnectorExecuteAction constructs ConnectorRewriteDriver — the path is reachable, not dormant." + } + ], + "corrected_multiplicity": "k-times-per-statement where k = number of bin-packed rewrite groups (per-partition bin-packing in RewriteDataFilePlanner.planAndOrganizeTasks can yield tens to hundreds of groups on large tables); each of the k calls is a full-table planFiles() at the same pinned snapshot, serial, on the rewrite_data_files execute path between the group writes and the commit.", + "mitigation_found": "None. No memoization in registerRewriteSourceFiles (per-call locals only), no path->DataFile cache on the transaction, iceberg manifest caching (io.manifest.cache-enabled) is off by default, and the driver does not union the groups before calling. Gate: only ALTER TABLE ... EXECUTE rewrite_data_files, but ungated within that command — every distributed rewrite with >1 group pays it.", + "fix_direction": "Hoist in the driver: replace the ConnectorRewriteDriver.java:143-144 loop with one registerRewriteSourceFiles call on the union of all groups' paths — the connector side needs zero changes (updateRewriteFiles already accumulates, and the fail-loud mismatch check works identically on union-size vs matched-size). Alternative (connector-side): memoize the single pinned-snapshot scan's path->DataFile map on the transaction on first call and serve subsequent calls from it. The driver-side union is simpler and keeps the one OCC-validating re-scan.", + "impact_estimate": "One rewrite_data_files statement performs G whole-table manifest scans in the registration step (plus 1 inherent planning scan) instead of 1+1. With M manifests and G groups that is G x O(M) serial remote reads plus O(G x total_files) CPU path-matching; on a table with thousands of manifests and G ~ 50-200 this stretches registration from seconds to minutes. Secondary effect: the serial scan loop sits inside the OCC window (after group writes, before commit), so the amplification also raises the probability of the 'table changed since planning' fail-loud abort and commit conflicts under concurrent writers." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "Every element of the candidate verified against the code. (1) Loop confirmed: ConnectorRewriteDriver.run() STEP 3 iterates `for (ConnectorRewriteGroup group : groups)` and calls connectorTx.registerRewriteSourceFiles(group.getDataFilePaths()) once per bin-packed group (ConnectorRewriteDriver.java:143-144). (2) Heavy op confirmed: each call runs table.newScan()[.useSnapshot(startingSnapshotId)].planFiles() — a full manifest-list + all-manifests remote read at the pinned OCC snapshot (IcebergConnectorTransaction.java:378-394), unfiltered, so its result is identical (loop-invariant) across all k calls; only the local `wanted` path set differs. (3) Mitigation hunt found NOTHING effective in default config: (a) no memoization inside the transaction — `wanted` and `matched` are method locals, no field caches the scan result across calls, and paths are not accumulated for a single lazy re-derive at commit; (b) the connector-level IcebergManifestCache exists but is wired ONLY into IcebergScanPlanProvider (IcebergConnector.java:162,590; consumers at IcebergScanPlanProvider.java:1738,1761) — the raw SDK planFiles() in registerRewriteSourceFiles bypasses it completely (a variant-C cache bypass layered on the variant-A loop); (c) the Iceberg SDK manifest content cache (io.manifest.cache-enabled) is default-OFF: IcebergCatalogFactory.java:68 DEFAULT_MANIFEST_CACHE_ENABLE=false, and appendManifestCacheProperties (119-134) only derives it true when the user explicitly sets io.manifest.cache-enabled or meta.cache.iceberg.manifest.enable — and even when enabled it only avoids re-fetching manifest bytes, each of the k scans still re-decodes Avro and re-evaluates every manifest entry (O(table-metadata) CPU per group); (d) no pass-down: RewriteDataFilePlanner already ran planFiles() once at the same current snapshot (RewriteDataFilePlanner.java:110-141) holding the exact DataFile objects, but IcebergProcedureOps.toConnectorRewriteGroup (IcebergProcedureOps.java:227-233) reduces them to String paths for the neutral SPI, forcing the connector to re-derive. (4) Gate is LIVE, not dormant: \"iceberg\" is in CatalogFactory.SPI_READY_TYPES (CatalogFactory.java:56-57), so this is the production path for every ALTER TABLE ... EXECUTE rewrite_data_files on any iceberg catalog. The class-level \"dormant until P6.6\" javadoc in IcebergConnectorTransaction is stale. Total per statement: 1 planning scan + k registration scans = k+1 whole-table metadata passes where 1 (arguably 2, given the OCC re-derive-at-pinned-snapshot design) would suffice.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java", + "line": 143, + "note": "STEP 3 per-group loop: `for (ConnectorRewriteGroup group : groups)` -> line 144 connectorTx.registerRewriteSourceFiles(group.getDataFilePaths()), one call per bin-packed group" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java", + "line": 383, + "note": "Each registerRewriteSourceFiles call runs scan.planFiles() (scan = table.newScan().useSnapshot(startingSnapshotId), lines 379-381) inside executeAuthenticated — full manifest-list + all-manifests remote read; `wanted`/`matched` (373-374) are locals, no cross-call memoization" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java", + "line": 398, + "note": "Mismatch check compares per-call wanted vs matched sizes — works identically on a single unioned call, so the trivial hoist needs no semantic change" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE = false; appendManifestCacheProperties (119-134) only sets io.manifest.cache-enabled=true on explicit user opt-in — SDK content cache does not mitigate in default config" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 162, + "note": "IcebergManifestCache instantiated here and passed only to IcebergScanPlanProvider (line 590); consumers are IcebergScanPlanProvider.java:1738/1761 — the raw SDK planFiles() in registerRewriteSourceFiles bypasses this cache entirely" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/rewrite/RewriteDataFilePlanner.java", + "line": 141, + "note": "Planning already ran tableScan.planFiles() once at the current snapshot holding the exact DataFile objects — the k registration scans re-derive information the connector side already computed" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergProcedureOps.java", + "line": 227, + "note": "toConnectorRewriteGroup strips DataFiles to raw String paths for the neutral SPI (228-230) — the pass-down that would avoid re-derivation is lost at this boundary" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java", + "line": 57, + "note": "\"iceberg\" is in SPI_READY_TYPES — the flip happened; ConnectorRewriteDriver is the live production path for ALTER TABLE ... EXECUTE rewrite_data_files (the 'dormant' javadoc in IcebergConnectorTransaction is stale)" + } + ], + "mitigation_found": "None effective in default configuration. Checked: (1) transaction-internal memoization — absent, scan result held only in method locals; (2) connector-level IcebergManifestCache (fe-connector-cache-backed) — exists but wired only into IcebergScanPlanProvider's split-planning path, bypassed by the raw table.newScan().planFiles() here; (3) Iceberg SDK io.manifest.cache-enabled content cache — off by default (DEFAULT_MANIFEST_CACHE_ENABLE=false, only derived true on explicit user opt-in via meta.cache.iceberg.manifest.enable or io.manifest.cache-enabled), and even when opted-in it is partial: it removes repeat manifest-byte fetches but each scan still re-decodes and re-evaluates all manifest entries (O(table-metadata) CPU per group), and does not cover the per-scan manifest-list read pattern or manifests above max-content-length; (4) upstream pass-down — the planner's DataFiles are deliberately reduced to String paths at the neutral SPI boundary, so nothing is passed down.", + "corrected_multiplicity": "k-times-per-statement, k = number of bin-packed rewrite groups (total whole-table planFiles passes per ALTER TABLE ... EXECUTE rewrite_data_files = k registration scans + 1 planning scan = k+1, vs the 2 the current OCC re-derive design needs or the 1 a full pass-down would need). Not per-split/per-query-planning — this is a maintenance-DDL path, so statement frequency is low, but k is largest exactly on the large fragmented tables compaction targets (10s-100s of groups), and the k scans run serially inside the open transaction window before commit.", + "impact_estimate": "O(k-groups-per-statement) x remote-IO (each scan = manifest-list + every manifest via FileIO, O(table metadata)) x trigger breadth: every iceberg catalog flavor, every ALTER TABLE ... EXECUTE rewrite_data_files with >=2 groups, default config (both manifest caches inactive on this path). On a table with thousands of manifests and k~50-200 groups this turns registration into minutes of serial redundant metadata IO between the group writes and the commit, which also widens the OCC window guarded by validateFromSnapshot(startingSnapshotId) and thereby raises the odds of a spurious commit conflict on a concurrently-written table. Severity per occurrence: high; occurrence rate: low (manual/scheduled compaction, not query planning).", + "fix_direction": "Hoist (minimal, engine-side): in ConnectorRewriteDriver STEP 3, union all groups' getDataFilePaths() into one Set and make ONE registerRewriteSourceFiles call — the SPI already takes Set<String>, updateRewriteFiles accumulates regardless, and the mismatch check works on totals; k+1 scans become 2. Equivalent connector-side memoize: have registerRewriteSourceFiles only accumulate paths and perform the single pinned-snapshot re-derive lazily at commit (first line of commitRewriteTxn). Deeper pass-down (optional, larger change): keep the planner's DataFile objects connector-side keyed by an opaque group id carried in ConnectorRewriteGroup, eliminating the re-scan entirely (k+1 -> 1), at the cost of stashing SDK objects across the neutral-SPI round trip — the hoist alone removes the amplification and is the surgical fix." + } + }, + { + "title": "One write statement loads the same iceberg table remotely 3-5 times (tableExists x2 + loadTable x2-3 + unconditional refresh) with no cache at any layer", + "variant": "B-chain-redundancy", + "heavy_op": "catalog.loadTable() — metastore RPC + metadata.json fetch/parse (IcebergCatalogOps.java:340-342, direct delegation, no CachingCatalog wrap anywhere in IcebergCatalogFactory/metastore-iceberg — grep-verified); plus catalogOps.tableExists remote check (IcebergConnectorMetadata.java:324); plus BaseTable.newTransaction()'s unconditional TableOperations.refresh() (documented at IcebergConnectorTransaction.java:189-191)", + "multiplicity": "k-times-per-query, k=3-5 per DML write: (1) MERGE/UPDATE distribution derivation: PhysicalIcebergMergeSink.getRequirePhysicalProperties (:161) -> getIcebergPartitioning (:198) -> buildInsertPartitionFieldsFromConnector -> metadata.getTableHandle (PhysicalIcebergMergeSink.java:307, remote exists) + getWritePartitioning (:316) -> resolveTable -> loadTable; (2) translation: PhysicalPlanTranslator.visitPhysicalConnectorTableSink -> metadata.getTableHandle (PhysicalPlanTranslator.java:675, remote exists) + getWriteSortColumns (:703) -> IcebergWritePlanProvider.getWriteSortColumns (:257) -> resolveTable (:689-702) -> loadTable; (3) sink bind: PluginDrivenTableSink.bindDataSink (:175) -> planWrite (:183) -> beginWrite -> catalogOps.loadTable (IcebergConnectorTransaction.java:208) + openTransaction refresh (:243-256); (4) EXPLAIN/plan render: PluginDrivenTableSink.getExplainString (:148) -> appendExplainInfo (IcebergWritePlanProvider.java:241) -> resolveTable -> loadTable.", + "entry_chain": "PhysicalIcebergMergeSink.getRequirePhysicalProperties (fe-core/.../physical/PhysicalIcebergMergeSink.java:161->198->307/316) AND PhysicalPlanTranslator.visitPhysicalConnectorTableSink (fe-core/.../translator/PhysicalPlanTranslator.java:675,703) AND PluginDrivenTableSink.bindDataSink (fe-core/.../planner/PluginDrivenTableSink.java:175) -> IcebergWritePlanProvider.resolveTable (IcebergWritePlanProvider.java:689-702) / IcebergConnectorTransaction.beginWrite (IcebergConnectorTransaction.java:208) -> CatalogBackedIcebergCatalogOps.loadTable (IcebergCatalogOps.java:340-342) -> remote catalog", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java", + "line": 689 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 341 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java", + "line": 703 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java", + "line": 316 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java", + "line": 208 + } + ], + "why_heavy": "Each loadTable is a metastore RPC (HMS getTable / REST loadTable) plus a FileIO read+JSON-parse of metadata.json (which grows with snapshot history — MBs on hot tables). All calls within one statement resolve the SAME db.table with no per-statement memoization: resolveTable builds a fresh load every call, CatalogBackedIcebergCatalogOps has no cache, and the factory does not wrap CachingCatalog. beginWrite then loads a 5th time and newTransaction refreshes once more milliseconds after that fresh load. Only beginWrite's load+refresh is inherent (OCC anchor); the getWriteSortColumns/getWritePartitioning/appendExplainInfo loads all read in-memory-after-load facets (sortOrder/spec/schema) that could come from one shared per-statement Table.", + "est_impact": "3-5 serial remote round-trips + metadata.json parses added to every iceberg DML's plan latency (per-query-constant, so it scales with write QPS); worst on REST catalogs with per-request session=user (each resolveTable also spins the delegated catalog path). Hoisting to one load per statement (e.g. resolve once onto the write handle / statement context) removes 2-4 of them.", + "gate": "getWritePartitioning leg only for UPDATE/MERGE with enable_iceberg_merge_partitioning session var on (PhysicalIcebergMergeSink.java:178 gates it); appendExplainInfo leg only when the plan explain string is rendered (EXPLAIN/profile); the other 3 (exists + sort-columns load + beginWrite load/refresh) run on every plugin-driven iceberg write.", + "confidence": "high", + "lens": "write-commit-path", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Independently re-traced every hop. (1) Heaviness holds: IcebergCatalogOps.loadTable (:340-342) is a bare catalog.loadTable delegation — metastore RPC + metadata.json fetch/parse per the SDK cost model; grep over fe-connector-iceberg and fe-connector-metastore-iceberg main sources finds zero CachingCatalog, and the only cache the factory wires (io.manifest.cache-enabled, IcebergCatalogFactory:120-131) covers manifest-file content in FileIO, not table loads. IcebergConnectorMetadata.getTableHandle does a remote tableExists per call (:324 -> IcebergCatalogOps:303-305). openTransaction (IcebergConnectorTransaction:243-256) goes through Transactions.newTransaction whose BaseTransaction ctor does an unconditional ops.refresh() — documented as a remote metastore call in the code's own javadoc (:188-191). (2) Multiplicity holds and is slightly understated: with no memoization at any layer (resolveTable at IcebergWritePlanProvider:689-702 loads fresh every call), every plugin-driven iceberg INSERT incurs 4 remote ops (translator exists at PhysicalPlanTranslator:675 + getWriteSortColumns load at :703->provider:257 + beginWrite load at PluginDrivenTableSink:175->provider:183->IcebergConnectorTransaction:208 + newTransaction refresh at :211); UPDATE/MERGE incurs 6 because the merge-partitioning leg's gate enable_iceberg_merge_partitioning DEFAULTS TO TRUE (SessionVariable.java:3829), firing PhysicalIcebergMergeSink:307 (exists) + :316 (getWritePartitioning->provider:286->load) from RequestPropertyDeriver:173/CostAndEnforcerJob:131; +1 load when the explain string renders (PluginDrivenTableSink:148->provider:241). (3) Cost is actually incurred: this is the mainline write path (visitPhysicalConnectorTableSink is the translation for all connector sinks; bindDataSink is the sink bind). Only beginWrite's load+refresh is inherent (OCC anchor; planWrite reuses transaction.getTable() at provider:184, no extra load). The sortColumns/partitioning/explain loads read in-memory-after-load facets (sortOrder/spec/schema) of the same db.table — classic variant-B single-chain redundancy: 2-4 redundant serial remote round-trips per DML statement.", + "corrected_multiplicity": "k-times-per-query on every iceberg DML: k=4 remote ops for plain INSERT (1 tableExists + 2 loadTable + 1 refresh), k=6 for UPDATE/MERGE because enable_iceberg_merge_partitioning defaults to TRUE (2 tableExists + 3 loadTable + 1 refresh), +1 loadTable when the plan explain string is rendered (EXPLAIN/profile). Only 1 loadTable + 1 refresh (beginWrite OCC anchor) is inherent; 2-4 remote ops per statement are redundant. Scales linearly with write QPS.", + "mitigation_found": "Only io.manifest.cache-enabled (derived to \"true\" from FE meta-cache settings, IcebergCatalogFactory.java:120-131) — a FileIO manifest-content cache that does NOT cover loadTable's metastore RPC or metadata.json fetch/parse. No CachingCatalog wrap anywhere in either module (grep-verified), no per-statement/per-session Table memo in resolveTable, ConnectorSession, StatementContext, or fe-connector-cache. BaseMetastoreTableOperations.refresh skips re-reading metadata.json when the location is unchanged, so the newTransaction refresh right after beginWrite's fresh load is usually 1 RPC without a metadata re-parse — a partial SDK-level softener for that one hop only.", + "fix_direction": "Hoist to one resolved Table per statement: memoize the loaded org.apache.iceberg.Table on the statement scope (e.g. cache it on/behind the IcebergTableHandle after the first resolveTable, or a per-queryId memo in IcebergWritePlanProvider keyed by db.table, mirroring the rewritableDeleteStash pattern already keyed by session.getQueryId()); have getWriteSortColumns/getWritePartitioning/appendExplainInfo read that shared Table, and let beginWrite keep its own fresh load+refresh as the OCC anchor (or conversely let it seed the memo). Separately, the translator/mergeSink getTableHandle tableExists RPCs can be skipped when fe-core already holds a resolved PluginDrivenExternalTable (the exists check re-verifies what table resolution already proved). Removes 2-4 remote ops per DML.", + "impact_estimate": "2-4 redundant serial metastore round-trips (HMS getTable / REST loadTable, most including a metadata.json fetch+JSON-parse that grows with snapshot history — MBs on hot tables) added to EVERY plugin-driven iceberg DML's plan latency; per-query-constant so total cost scales with write QPS; worst on REST catalogs with per-request session=user and on high-snapshot-count tables. Not per-split, so bounded per statement — moderate severity, high breadth (all iceberg writes).", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 341, + "note": "loadTable is a bare catalog.loadTable(toTableIdentifier(...)) delegation — remote metastore RPC + metadata.json fetch, no cache; tableExists at :303-305 is likewise a bare catalog.tableExists delegation" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java", + "line": 689, + "note": "resolveTable builds ops fresh via catalogOpsResolver.apply(session) and calls ops.loadTable every invocation (:693/:697) — no memoization; called from appendExplainInfo:241, getWriteSortColumns:257, getWritePartitioning:286" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java", + "line": 183, + "note": "planWrite -> transaction.beginWrite then reuses transaction.getTable() at :184 — bind leg is exactly 1 loadTable + 1 refresh (this part is the inherent OCC anchor)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java", + "line": 208, + "note": "beginWrite does catalogOps.loadTable(db, tableName) then openTransaction(loaded) at :211; javadoc :188-191 states BaseTable.newTransaction() issues an unconditional TableOperations.refresh() (remote metastore call); openTransaction :243-256 routes both arms through Transactions.newTransaction (BaseTransaction ctor refreshes)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 324, + "note": "getTableHandle does context.executeAuthenticated(() -> catalogOps.tableExists(dbName, tableName)) — a remote existence RPC on every call, no cache; handle build itself is pure" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java", + "line": 675, + "note": "visitPhysicalConnectorTableSink: metadata.getTableHandle (remote exists) at :675-676, then writePlanProvider.getWriteSortColumns at :703 (-> resolveTable -> loadTable) — runs on every plugin-driven connector INSERT/DML translation" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java", + "line": 316, + "note": "getRequirePhysicalProperties (:161) gated at :178 by isEnableIcebergMergePartitioning -> buildInsertPartitionFieldsFromConnector: metadata.getTableHandle at :307-308 (remote exists) + writePlanProvider.getWritePartitioning at :316 (-> resolveTable -> loadTable); invoked from RequestPropertyDeriver.java:173 inside CostAndEnforcerJob.java:131" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java", + "line": 3829, + "note": "enableIcebergMergePartitioning = true BY DEFAULT — the merge-partitioning leg (exists + loadTable) runs on every default-config UPDATE/MERGE, making the gate stronger than the finding claimed" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java", + "line": 175, + "note": "bindDataSink -> writePlanProvider.planWrite (the beginWrite load+refresh leg); getExplainString :148 -> appendExplainInfo (-> resolveTable -> loadTable), fires only when the explain string is rendered as the finding's gate stated" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 120, + "note": "only cache wired at construction is io.manifest.cache-enabled (:120-131, manifest-file content in FileIO) — does not cover loadTable's metastore RPC or metadata.json; grep over both iceberg connector modules finds zero CachingCatalog" + } + ] + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "Every hop of the candidate chain checks out against the code, and an exhaustive mitigation hunt found no layer that dedupes the loads in the default configuration. (1) IcebergWritePlanProvider.resolveTable (:689-702) performs a fresh ops.loadTable on every call with no memo; its three callers (appendExplainInfo :241, getWriteSortColumns :257, getWritePartitioning :286) each read only in-memory-after-load facets (sortOrder/spec/schema) of the SAME table. (2) CatalogBackedIcebergCatalogOps.loadTable/tableExists (:340-342/:303-305) delegate directly to the raw SDK catalog — tree-wide grep confirms no CachingCatalog wrap anywhere in fe-connector (only a paimon comment mentions it). (3) IcebergConnectorMetadata.getTableHandle (:317-332) does a remote tableExists per call; on HMS-backed catalogs the SDK-default tableExists is a full loadTable attempt (metastore RPC + metadata.json fetch/parse), on REST a HEAD RPC. (4) The merge-sink derivation leg runs BY DEFAULT: both gates — enableIcebergMergePartitioning (SessionVariable.java:3829) and enableStrictConsistencyDml (:1612, checked at RequestPropertyDeriver.java:170) — default true. I verified it runs exactly once per statement, not k times: RequestPropertyDeriver:173 is the only invoking site; ShuffleKeyPruner overrides only connector/olap/dictionary sink visitors (:219/:242/:260) and SinkVisitor.java:168 routes the merge sink to the generic visitPhysicalTableSink default. (5) beginWrite (IcebergConnectorTransaction.java:193-222) loads a final time and newTransaction issues an unconditional TableOperations.refresh (:188-191 doc, :243-256 impl) — this one is the inherent OCC anchor, correctly excluded. Mitigations searched and ruled out: no Table-object cache in fe-connector-iceberg, fe-connector-metastore-iceberg, or fe-connector-cache (grep for Cache<...,Table>/MetaCacheEntry: zero hits); IcebergLatestSnapshotCache caches only (snapshotId, schemaId) pins (IcebergLatestSnapshotCache.java:54-67); IcebergManifestCache caches manifest bytes and is default-off; the meta.cache.iceberg.table.ttl-second knob feeds only the snapshot-pin + schema caches (IcebergConnector.java:188-190), never loadTable; IcebergTableHandle is a pure serializable handle with no Table field; applySnapshot (IcebergConnectorMetadata.java:1661-1673) and applyMvccSnapshotPin (PluginDrivenScanNode.java:859-868) are pure in-memory. Partial mitigations that exist but do not cover the redundancy: planWrite reuses transaction.getTable() from beginWrite (IcebergWritePlanProvider.java:184 — so plan+begin share ONE load), the distributed-rewrite writeStarted guard shares one beginWrite across groups (IcebergConnectorTransaction.java:195-201), and the plain-INSERT sink's getRequirePhysicalProperties (PhysicalConnectorTableSink.java:161-253) is pure in-memory so INSERT has no derivation leg. Net: INSERT = 2 redundant remote round-trips (exists + sort-columns load) on top of the inherent load+refresh; UPDATE/MERGE = 3 redundant (derivation exists + derivation load + translator exists); EXPLAIN adds 1 more (appendExplainInfo load, PluginDrivenTableSink.java:148 builds a transient handle and calls it before planWrite). One refinement to the candidate: it under-specified that the derivation leg has a second default-true gate (enableStrictConsistencyDml) and its suggestion that k could grow via repeated getRequirePhysicalProperties calls is wrong — it is called once.", + "corrected_multiplicity": "k-times-per-query (per DML statement), verified k: INSERT = 4 remote round-trips (2 redundant: translator tableExists + getWriteSortColumns loadTable; beginWrite loadTable + unconditional newTransaction refresh are the inherent OCC anchor); UPDATE/MERGE = 5 (3 redundant: merge-sink derivation tableExists + getWritePartitioning loadTable — runs exactly once, both gating session vars default true — plus translator tableExists); +1 loadTable whenever the plan explain text is rendered (EXPLAIN/profile). Not per-split/per-partition — scales linearly with write QPS.", + "mitigation_found": "None that prevents the repeated remote cost in the default configuration. Partial mechanisms found and verified insufficient: (1) planWrite reuses transaction.getTable() from beginWrite (IcebergWritePlanProvider.java:184), so sink-plan build + transaction begin share one load; (2) distributed-rewrite groups share one beginWrite via the writeStarted guard (IcebergConnectorTransaction.java:195-201); (3) IcebergLatestSnapshotCache caches only (snapshotId, schemaId) pins and IcebergManifestCache caches manifest content (consumed only when meta.cache.iceberg.manifest.enable is set, default off) — neither serves loadTable/tableExists; (4) plain-INSERT distribution derivation (PhysicalConnectorTableSink.getRequirePhysicalProperties) is pure in-memory. Ruled out: no CachingCatalog wrap (tree-wide grep), no Table-object cache in fe-connector-iceberg / fe-connector-metastore-iceberg / fe-connector-cache, IcebergTableHandle carries no Table, meta.cache.iceberg.table.ttl-second governs only snapshot-pin/schema caches, applySnapshot/applyMvccSnapshotPin pure in-memory.", + "impact_estimate": "O(k-per-query) x remote-IO x every plugin-driven iceberg DML write on every catalog flavor. k_redundant = 2 serial remote round-trips per INSERT, 3 per UPDATE/MERGE (+1 under EXPLAIN/plan render). Each redundant op is a metastore RPC plus (for loadTable, and for tableExists on HMS-backed catalogs where the SDK default implements exists as a full load attempt) a FileIO fetch + JSON parse of metadata.json, which grows with snapshot history (MBs on hot tables). REST catalogs pay a full loadTable for the sort-columns/partitioning legs and a HEAD per exists; with iceberg.rest.session=user each resolveTable additionally goes through the per-request delegated catalog path. Latency is additive and serial on the statement's planning critical path, so total added cluster load scales with write QPS. Merge/update leg gated by enableIcebergMergePartitioning AND enableStrictConsistencyDml — both default TRUE, so it fires by default.", + "fix_direction": "Hoist + pass-down, per-statement: (a) in PhysicalPlanTranslator.visitPhysicalConnectorTableSink / buildPluginRowLevelDmlSink, drop the separate remote exists check by resolving once and deriving existence from the load (or stash the handle resolved during analysis/derivation on the StatementContext and reuse it); (b) memoize the loaded SDK Table per statement — e.g. a queryId-keyed short-lived Table memo in the connector (mirroring IcebergRewritableDeleteStash's per-query stash pattern) or carried via ConnectorSession — so getWritePartitioning, getWriteSortColumns, appendExplainInfo and beginWrite share ONE load; (c) alternatively defer sort-column/partitioning derivation to planWrite where transaction.getTable() is already loaded, threading the results back on the write handle (params-level pass-down). Wrapping the SDK catalog in CachingCatalog would also dedupe but changes cross-statement freshness semantics; a per-statement memo is the surgical fix. beginWrite's load + newTransaction refresh must stay (OCC anchor).", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java", + "line": 689, + "note": "resolveTable builds a fresh ops.loadTable per call (:691-697), no memo field, no cache; callers at :241 (appendExplainInfo), :257 (getWriteSortColumns), :286 (getWritePartitioning) each re-load the same table to read in-memory facets (sortOrder/spec/schema)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "CatalogBackedIcebergCatalogOps.loadTable = direct catalog.loadTable delegation (:341); tableExists = direct catalog.tableExists (:303-304). Tree-wide grep: no CachingCatalog wrap anywhere in fe-connector main sources" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 324, + "note": "getTableHandle does a remote catalogOps.tableExists per call inside executeAuthenticated; comment confirms only the handle build is pure. On HMS-backed catalogs the SDK-default tableExists is a full loadTable attempt (metadata.json fetch+parse); REST uses a HEAD RPC" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java", + "line": 675, + "note": "INSERT leg: getTableHandle (remote exists) at :675, then getWriteSortColumns at :703 (second remote load via resolveTable). Row-level DML leg does its own getTableHandle at :623. applyMvccSnapshotPin (:695/:636) is pure in-memory (verified applySnapshot at IcebergConnectorMetadata.java:1661-1673)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java", + "line": 307, + "note": "getRequirePhysicalProperties (:161) -> getIcebergPartitioning (:198) -> buildInsertPartitionFieldsFromConnector: metadata.getTableHandle (:307-308, remote exists) + getWritePartitioning (:316, remote loadTable). Gate at :178 (enableIcebergMergePartitioning) defaults TRUE (SessionVariable.java:3829); second gate enableStrictConsistencyDml (RequestPropertyDeriver.java:170) defaults TRUE (SessionVariable.java:1612)" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/properties/RequestPropertyDeriver.java", + "line": 173, + "note": "Sole invoker of the merge sink's getRequirePhysicalProperties — ShuffleKeyPruner overrides only olap/connector/dictionary sink visitors (:219/:242/:260) and SinkVisitor.java:168 default-routes visitPhysicalIcebergMergeSink to visitPhysicalTableSink, so the remote derivation runs exactly once per statement (not k times)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java", + "line": 208, + "note": "beginWrite loads the table again (:208) and openTransaction/newTransaction issues an unconditional TableOperations.refresh (:188-191 doc, :243-256 impl) — the inherent OCC anchor. writeStarted guard (:195-201) shares one beginWrite only across distributed-rewrite groups; planWrite reuses transaction.getTable() (IcebergWritePlanProvider.java:184) so plan+begin share this one load" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 189, + "note": "The only per-catalog caches are IcebergLatestSnapshotCache (snapshotId+schemaId pins only, IcebergLatestSnapshotCache.java:54-67) fed by meta.cache.iceberg.table.ttl-second, and the path-keyed IcebergManifestCache (default off). Grep for Table-valued caches (Cache<..,Table>/MetaCacheEntry) across fe-connector-iceberg, fe-connector-metastore-iceberg, fe-connector-cache: zero hits. IcebergTableHandle (IcebergTableHandle.java:52-103) is a pure data handle, no Table field" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSink.java", + "line": 162, + "note": "Counter-check: the plain-INSERT sink's getRequirePhysicalProperties (:162-253) is pure in-memory (capability bits + cached partition columns) — the remote derivation leg exists only on the UPDATE/MERGE sink, so INSERT's redundant count is 2, not 3" + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/planner/PluginDrivenTableSink.java", + "line": 148, + "note": "getExplainString (:134-150) builds a transient write handle and calls appendExplainInfo -> resolveTable -> loadTable before planWrite has run — the +1 EXPLAIN/plan-render load; bindDataSink (:161-177) -> planWrite (:175) is the begin-write leg" + } + ] + } + }, + { + "title": "expire_snapshots builds its stats map by reading EVERY snapshot's delete manifests with no manifest-level dedup (S x M remote reads)", + "variant": "A-loop-amplification", + "heavy_op": "snapshot.deleteManifests(io) (manifest-LIST read per snapshot) + ManifestFiles.readDeleteManifest (full delete-manifest avro read per manifest per snapshot) — IcebergExpireSnapshotsAction.java:274-287", + "multiplicity": "per-snapshot x per-delete-manifest over the ENTIRE table history: outer loop `for (Snapshot snapshot : icebergTable.snapshots())` (IcebergExpireSnapshotsAction.java:274), inner loop over that snapshot's delete manifests (:279-287). Consecutive snapshots overwhelmingly share the same manifest files, but there is no visited-manifest set — the same manifest is re-read for every snapshot that references it (dedup happens only per DeleteFile path AFTER the read, :283).", + "entry_chain": "ALTER TABLE ... EXECUTE expire_snapshots -> IcebergProcedureOps.execute (IcebergProcedureOps.java:116) -> runInAuthScope -> BaseIcebergAction.execute -> IcebergExpireSnapshotsAction.executeAction (IcebergExpireSnapshotsAction.java:150) -> buildDeleteFileContentMap (:168-169 call, body :271-293) -> per-snapshot deleteManifests(io) (:275) -> per-manifest ManifestFiles.readDeleteManifest (:280)", + "files": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 275 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 280 + } + ], + "why_heavy": "Each iteration is remote IO (manifest-list fetch + delete-manifest avro decode). With S snapshots each referencing ~M largely-shared delete manifests, the loop performs O(S x M) manifest reads where O(distinct manifests) would suffice — and the entire map exists ONLY to classify the deleteWith callback's counter columns (:215-231, position vs equality delete counts); files that expire never come from snapshots that survive, so most of the history walked is irrelevant to the result.", + "est_impact": "On a long-history table (hundreds/thousands of snapshots — precisely the tables one runs expire_snapshots on), the pre-pass alone can take minutes of serial remote IO before ExpireSnapshots.commit() even starts, and it holds the whole path->content map in FE memory. A visited-manifest-path Set (or restricting the walk to the snapshots being expired) reduces it to O(distinct manifests of expired snapshots).", + "gate": "Only ALTER TABLE ... EXECUTE expire_snapshots (maintenance procedure, not query planning); runs unconditionally on every invocation regardless of arguments.", + "confidence": "high", + "lens": "write-commit-path", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Re-derived independently from the code. (1) Heaviness holds: line 275 `snapshot.deleteManifests(icebergTable.io())` forces the SDK to read that snapshot's manifest-LIST file via FileIO (BaseSnapshot.cacheManifests — remote IO, one per snapshot, since each snapshot has its own manifest-list file), and line 280 `ManifestFiles.readDeleteManifest(manifest, io, specs)` opens and fully iterates the delete-manifest avro — remote IO per call, squarely in the given SDK cost model. The only in-memory call in the loop is `icebergTable.specs()`. (2) Multiplicity holds as claimed or worse: the outer loop (line 274) walks EVERY snapshot in table history; `Snapshot.deleteManifests` returns ALL delete manifests visible at that snapshot (the full live set from its manifest list, not the delta), so consecutive snapshots share nearly all manifests, and there is no visited-manifest-path set — the same manifest file is fully re-read for each snapshot referencing it. The only dedup is per-DeleteFile `putIfAbsent` (line 283) which runs AFTER the remote read, so it prevents no IO. Total = O(S) manifest-list reads + O(S x M) delete-manifest reads where O(distinct manifests) suffices. Note it is even worse than claimed in one dimension: even for an append-only table with zero delete files, the per-snapshot manifest-LIST read at line 275 still occurs for all S snapshots (the emptiness check at 276 requires the list to have been read). (3) Cost actually incurred: the action is registered (IcebergExecuteActionFactory.java:51,81) and dispatched via IcebergProcedureOps.execute (:116) -> runInAuthScope (:175/:179 `action.execute(ops.loadTable(...))`) -> BaseIcebergAction.execute (:109-110) -> executeAction, which calls buildDeleteFileContentMap unconditionally (IcebergExpireSnapshotsAction.java:168-169) before ExpireSnapshots is even configured, regardless of arguments. The map's sole consumer is the deleteWith counter classification (:215-231). The one potential mitigation — the Iceberg SDK ContentCache via `io.manifest.cache-enabled` — is derived from `meta.cache.iceberg.manifest.enable` whose default is false (IcebergCatalogFactory.java:68 DEFAULT_MANIFEST_CACHE_ENABLE=false; derivation :119-133), so by default every readDeleteManifest is an uncached remote read; per the audit rules a disabled-by-default cache does not refute. Gate as declared: maintenance procedure (ALTER TABLE ... EXECUTE expire_snapshots), not query planning — matches problem-class variant A (loop amplification) with an honest gate note.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 274, + "note": "Outer loop `for (Snapshot snapshot : icebergTable.snapshots())` — walks ENTIRE table snapshot history, not just snapshots being expired." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 275, + "note": "`snapshot.deleteManifests(icebergTable.io())` — per-snapshot manifest-LIST remote read (SDK BaseSnapshot.cacheManifests); happens for all S snapshots even on append-only tables." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 280, + "note": "`ManifestFiles.readDeleteManifest(manifest, io, specs)` — full delete-manifest avro remote read per (snapshot, manifest) pair; no visited-manifest-path set, so shared manifests are re-read for every snapshot referencing them." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 283, + "note": "Dedup is only `putIfAbsent` per DeleteFile path AFTER the manifest has already been read — prevents zero IO." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 168, + "note": "buildDeleteFileContentMap(icebergTable) called unconditionally at the top of executeAction on every invocation, regardless of arguments." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 215, + "note": "Sole consumer of the map: deleteWith callback classifying deleted paths into position/equality delete counters (:215-231)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE = false — SDK `io.manifest.cache-enabled` derivation (:119-133) leaves the Iceberg ContentCache OFF by default, so readDeleteManifest re-reads are real remote IO unless the user explicitly enables the cache." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExecuteActionFactory.java", + "line": 81, + "note": "Action is registered/reachable: factory constructs IcebergExpireSnapshotsAction for \"expire_snapshots\" (constant at :51)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergProcedureOps.java", + "line": 175, + "note": "Entry chain confirmed: execute(:116) -> runInAuthScope(:168) -> action.execute(ops.loadTable(...)) at :175 (non-auth) / :179 (executeAuthenticated)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/BaseIcebergAction.java", + "line": 110, + "note": "BaseIcebergAction.execute (:109) invokes executeAction exactly once per procedure invocation — the S x M blowup is entirely inside that single call." + } + ], + "corrected_multiplicity": "As claimed, plus one extra dimension: per-snapshot x per-live-delete-manifest (S x M) full delete-manifest reads with no manifest-level dedup, AND O(S) manifest-list remote reads that occur for every snapshot even when the table has zero delete files (append-only tables still pay S manifest-list fetches). k=1 invocation per EXECUTE statement; the amplification is entirely within that one call.", + "fix_direction": "Add a visited-manifest-path Set (skip ManifestFiles.readDeleteManifest for already-read manifest.path()) — reduces delete-manifest reads from O(S x M) to O(distinct manifests). Better: restrict the walk to snapshots that will actually be expired (or compute the classification from a before/after metadata diff like Spark's ExpireSnapshotsSparkAction), which also eliminates the O(S) manifest-list reads for surviving history. Alternatively classify by reading only manifests of expired snapshots after commit determines them.", + "impact_estimate": "On the long-history tables expire_snapshots targets (hundreds to thousands of snapshots), the pre-pass performs S manifest-list fetches plus roughly S x M serial delete-manifest avro reads before ExpireSnapshots.commit() starts — minutes of serial remote IO and an FE-memory path->content map, where O(distinct delete manifests) reads (typically 100-1000x fewer) would produce the identical map. Gated to the maintenance procedure, not query planning, and mitigated only if the user explicitly enables the default-off iceberg manifest content cache.", + "mitigation_found": "Only one candidate mitigation exists and it is off by default: the Iceberg SDK ContentCache (`io.manifest.cache-enabled`), derived from `meta.cache.iceberg.manifest.enable` with DEFAULT_MANIFEST_CACHE_ENABLE=false (IcebergCatalogFactory.java:68, 119-133). When a user explicitly enables it, repeated reads of the same delete manifest (up to the cache's per-file/total byte caps) would be served from memory, blunting the S x M term but not the O(S) manifest-list reads. Doris's own IcebergManifestCache is scan-planning-only and not consulted here. No hoist/memoize exists in the action itself." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The candidate is accurate and no mitigation covers this path in the default configuration. buildDeleteFileContentMap (IcebergExpireSnapshotsAction.java:271-293) walks EVERY snapshot in table history (:274), performs one remote manifest-list read per snapshot via snapshot.deleteManifests(io) (:275 — the SDK's BaseSnapshot caches only within a single Snapshot object, no cross-snapshot sharing), and fully re-reads every referenced delete manifest via ManifestFiles.readDeleteManifest (:280) with no visited-manifest-path set; the only dedup is per-DeleteFile putIfAbsent AFTER the read (:283). Since manifest lists carry existing delete manifests forward across snapshots, the same immutable manifest file is re-fetched and re-decoded once per referencing snapshot: O(S x M) remote reads where O(distinct manifests) suffices. Two candidate mitigations were found and both fail: (1) the per-catalog IcebergManifestCache (IcebergConnector.java:162) caches EXACTLY this read (its loader loadDeleteFiles at IcebergManifestCache.java:209-214 issues the same ManifestFiles.readDeleteManifest, keyed by manifest path+content, immutable values, no TTL) — but it is consumed only by IcebergScanPlanProvider's manifest-level planning path; the action calls the SDK directly with raw icebergTable.io() and never consults it (cache-bypass, variant C); moreover even that consumer is gated by meta.cache.iceberg.manifest.enable, default off. (2) The Iceberg SDK ContentCache: IcebergCatalogFactory.appendManifestCacheProperties (:119-134) sets io.manifest.cache-enabled=true only when the user enables meta.cache.iceberg.manifest.enable (DEFAULT_MANIFEST_CACHE_ENABLE=false at :68, doc 'Default-disabled' at :117), so it is off by default; even when on it is partial — the per-snapshot manifest-LIST reads at :275 bypass ManifestFiles.newInputFile so are never content-cached, and avro decode still repeats S x M times on cached bytes. Entry chain verified: IcebergProcedureOps.execute (:116) -> runInAuthScope (:125, body :168) -> action.execute(ops.loadTable(...)) (:175/:179) -> BaseIcebergAction.execute (:109-110) -> executeAction (IcebergExpireSnapshotsAction.java:150) -> buildDeleteFileContentMap (:168-169). The pre-pass runs unconditionally on every expire_snapshots invocation, before ExpireSnapshots is even configured, and the map exists only to classify deleteWith callback paths into position/equality counters (:215-231). One scoping note: this is a maintenance-procedure path (ALTER TABLE ... EXECUTE expire_snapshots), not query planning, so the multiplier is per-invocation table history, not QPS.", + "corrected_multiplicity": "per-snapshot x per-delete-manifest over the ENTIRE table history per invocation: O(S) remote manifest-list reads (one per snapshot, :275) plus O(S x M) full delete-manifest avro reads (:280) where consecutive snapshots overwhelmingly share manifests, so O(distinct delete manifests) would suffice — amplification factor approaches S on stable delete-manifest sets. Finding's stated multiplicity is correct as cited. For tables with no delete files the inner loop vanishes but the O(S) manifest-list reads remain.", + "mitigation_found": "Two partial/off-by-default mechanisms, neither effective here: (1) per-catalog IcebergManifestCache (IcebergConnector.java:162; delete-manifest loader IcebergManifestCache.java:209-214) caches exactly this read keyed by manifest path+content, but is wired only into IcebergScanPlanProvider's manifest-level planning path — the action bypasses it entirely by calling ManifestFiles.readDeleteManifest with raw table.io(); and its consumer gate meta.cache.iceberg.manifest.enable defaults to off. (2) Iceberg SDK ContentCache via io.manifest.cache-enabled, derived in IcebergCatalogFactory.appendManifestCacheProperties (:119-134) from the same meta.cache.iceberg.manifest.enable flag — default false (:68, :117); even when enabled it does not cover the per-snapshot manifest-list reads (:275) and only avoids re-fetching bytes, not the repeated S x M avro decodes. No upstream compute-once/pass-down exists: the map is built solely for the deleteWith counter classification (:215-231).", + "impact_estimate": "O(snapshots x delete-manifests) x remote-IO x every ALTER TABLE ... EXECUTE expire_snapshots invocation on any iceberg table (unconditional pre-pass, no argument short-circuits it). Worst exactly where the procedure is most needed: long-history tables (hundreds-thousands of snapshots) with v2 delete files — serial remote reads of largely-identical manifests can dominate wall time (minutes) before ExpireSnapshots.commit() starts, plus the whole path->FileContent map held in FE memory. Not on the query-planning hot path (admin-triggered maintenance), so per-invocation severity is high but invocation frequency is low.", + "fix_direction": "memoize/hoist within the pre-pass: add a visited-manifest-path Set keyed by ManifestFile.path() before the read at :280 (manifest content is immutable per path), collapsing cost to O(distinct delete manifests); better, route the read through the existing per-catalog IcebergManifestCache (loader IcebergManifestCache.java:209 is byte-identical to this read) so cross-invocation and cross-query reuse also apply; additionally the walk could be restricted to snapshots that will actually expire (files surviving in retained snapshots are never passed to deleteWith), reducing the outer O(S) manifest-list reads.", + "evidence": [ + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 274, + "note": "outer loop over ALL table-history snapshots: for (Snapshot snapshot : icebergTable.snapshots())" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 275, + "note": "snapshot.deleteManifests(icebergTable.io()) — one remote manifest-list read per snapshot; SDK caches only per-Snapshot-object" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 280, + "note": "ManifestFiles.readDeleteManifest with raw icebergTable.io() — full remote avro read per manifest per referencing snapshot, no visited-manifest set, bypasses IcebergManifestCache" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 283, + "note": "dedup is putIfAbsent per DeleteFile path AFTER the read — confirms no manifest-level dedup" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 168, + "note": "buildDeleteFileContentMap invoked unconditionally at the top of executeAction, before any ExpireSnapshots configuration" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java", + "line": 215, + "note": "sole consumer of the map: deleteWith callback classifying deleted paths into position/equality delete counters (:215-231)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergProcedureOps.java", + "line": 116, + "note": "entry: execute -> runInAuthScope (:125); action.execute(ops.loadTable(...)) at :175/:179 — table load itself is once per procedure (not the amplification)" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 162, + "note": "per-catalog IcebergManifestCache exists but is passed only to IcebergScanPlanProvider — the procedure path never receives or consults it" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java", + "line": 209, + "note": "cache loader loadDeleteFiles performs the identical ManifestFiles.readDeleteManifest(manifest, table.io(), table.specs()) — proving the repeated read is cacheable and a suitable memoization layer already exists, just unwired here" + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE = false — SDK io.manifest.cache-enabled derivation is default-disabled (derivation logic :119-134), so the Iceberg ContentCache does not dedupe these reads in default config" + } + ] + } + }, + { + "title": "One MTMV refresh flow re-materializes the same latest partition view k~4-6 times before the per-partition loops (no refresh-scoped pin)", + "variant": "B-chain-redundancy", + "heavy_op": "materializeLatest: catalogOps.loadTable + PARTITIONS metadata-table planFiles (buildMvccPartitionView for range-eligible tables, listPartitions/loadRawPartitions for identity-partitioned) per call", + "multiplicity": "k-times-per-refresh: (1) isValidRelatedTable per pct table (MTMVTask.java:221 -> PluginDrivenMvccExternalTable.java:787 materializeLatest, cost acknowledged in its own comment); (2) alignMvPartition -> generateRelatedPartitionDescs -> MTMVRelatedPartitionDescInitGenerator.getAndCopyPartitionItems (MTMVTask.java:228 -> MTMVPartitionUtil.java:136-139,187-193 -> MTMVRelatedPartitionDescInitGenerator.java:43); (3) MTMVRefreshContext.buildContext -> MTMV.calculatePartitionMappings -> generateRelatedPartitionDescs AGAIN (MTMVTask.java:243 -> MTMVRefreshContext.java:61-64 -> MTMV.java:533-541); (4) +1 per RollUp/DateTrunc generator getPartitionType/getPartitionColumnType call (MTMVPartitionExprDateTrunc.java:77, MTMVRelatedPartitionDescRollUpGenerator.java:61, MTMVPartitionUtil.java:626) — each an independent getOrMaterialize(empty)", + "entry_chain": "MTMVTask.run (MTMVTask.java:221/228/243) -> {isValidRelatedTable (PluginDrivenMvccExternalTable.java:787) | alignMvPartition (MTMVPartitionUtil.java:136-139) -> generateRelatedPartitionDescs (MTMVPartitionUtil.java:187-193) -> MTMVRelatedPartitionDescInitGenerator.apply (MTMVRelatedPartitionDescInitGenerator.java:43) -> getAndCopyPartitionItems (PluginDrivenMvccExternalTable.java:585) -> getNameToPartitionItems (PluginDrivenMvccExternalTable.java:580-582) | MTMVRefreshContext.buildContext (MTMVRefreshContext.java:61-64) -> MTMV.calculatePartitionMappings (MTMV.java:533-541) -> generateRelatedPartitionDescs (same path)} -> getOrMaterialize(empty) (PluginDrivenMvccExternalTable.java:114-119) -> materializeLatest (PluginDrivenMvccExternalTable.java:126-193) -> loadTable + PARTITIONS planFiles (IcebergConnectorMetadata.java:1453 / IcebergPartitionUtils.java:712)", + "files": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 787 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescInitGenerator.java", + "line": 43 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java", + "line": 539 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java", + "line": 228 + } + ], + "why_heavy": "Each hop independently pays a remote catalog loadTable plus a full PARTITIONS metadata scan for the SAME table at (almost certainly) the same snapshot within a single refresh flow lasting seconds. The information (latest partition view) is invariant across the flow, but there is no refresh-context-scoped MvccSnapshot pin: MTMVRefreshContext caches only MTMVSnapshotIf table snapshots (MTMVRefreshContext.java:35), and each generic accessor (isValidRelatedTable, getAndCopyPartitionItems, getPartitionType) re-materializes from scratch. alignMvPartition and buildContext even compute generateRelatedPartitionDescs twice back-to-back.", + "est_impact": "4-6 redundant (loadTable + PARTITIONS planFiles) rounds per refresh per external pct table — hundreds of ms to seconds each on manifest-heavy tables — plus a snapshot-skew hazard between the enumeration points. Dwarfed by finding 1 in absolute cost but independent: fixing 1 (pin during loops) does not remove these k duplicate materializations.", + "gate": "MTMV whose pct/base table is a plugin MVCC external table (iceberg/paimon); every scheduled or manual refresh task pays it, even a no-op refresh.", + "confidence": "high", + "lens": "stats-partitions-freshness", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Both heaviness and multiplicity hold as claimed, and the EXPR/date_trunc case is WORSE than claimed. Multiplicity: within one MTMVTask.run, materializeLatest() runs (1) via isValidRelatedTable (MTMVTask.java:221 -> PluginDrivenMvccExternalTable.java:787; its own comment counts only this call site as 'once per refresh'), (2) via alignMvPartition -> generateRelatedPartitionDescs -> InitGenerator.getAndCopyPartitionItems (MTMVTask.java:228 -> MTMVPartitionUtil.java:138,192-193 -> MTMVRelatedPartitionDescInitGenerator.java:43 -> PluginDrivenMvccExternalTable.java:585/580 -> getOrMaterialize:114-118), and (3) via buildContext -> calculatePartitionMappings -> generateRelatedPartitionDescs AGAIN (MTMVTask.java:243 -> MTMVRefreshContext.java:63 -> MTMV.java:539-540). For EXPR (date_trunc) MTMVs each generateRelatedPartitionDescs pass ADDS: one getPartitionType (MTMVRelatedPartitionDescRollUpGenerator.java:61) plus one getPartitionColumnType PER PARTITION DESC inside the rollUpRange loop (RollUpGenerator:145-147 -> MTMVPartitionExprDateTrunc.java:134 -> MTMVPartitionUtil.java:626 -> getPartitionColumns(empty) -> getOrMaterialize) — per-partition multiplicity, not '+1' as claimed. No memoization exists at any layer: MvccUtil.getSnapshotFromContext (MvccUtil.java:35-45) delegates to StatementContext.getSnapshot (StatementContext.java:1015-1034), a passive map lookup with no lazy load; at MTMVTask lines 221/228/243 the thread's ConnectContext has no populated StatementContext (MTMVPlanUtil's temp contexts at MTMVPlanUtil.java:222-230/259-267 are closed and restored before returning), so every accessor gets Optional.empty() and re-materializes. MTMVRefreshContext.java:35 caches only MTMVSnapshotIf table snapshots, never the partition view. Heaviness: each iceberg materializeLatest pays a remote tableExists RPC (PluginDrivenExternalTable.java:116-119 -> IcebergConnectorMetadata.java:324), an UNCACHED catalogOps.loadTable (raw SDK delegation, IcebergCatalogOps.java:340-341; no CachingCatalog wrap anywhere in fe-connector-iceberg/metastore-iceberg), and a PARTITIONS metadata-table newScan().useSnapshot(id).planFiles() (IcebergConnectorMetadata.java:1453-1454 -> IcebergPartitionUtils.java:710-712) — remote manifest-list+manifest IO per the SDK cost model. All k calls happen even for a NOT_REFRESH no-op (the decision at MTMVTask.java:248-250 comes after all of them). Minor citation correction only: MTMVPartitionExprDateTrunc.java:77 is in analyze() (MTMV creation), not the refresh path; the refresh-path DateTrunc hop is :134-135, which is stronger.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java", + "line": 221, + "note": "isValidRelatedTable per pct table inside run(); alignMvPartition at :228 and buildContext at :243 follow in the same flow; NOT_REFRESH short-circuit only at :248-250, after all three." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 787, + "note": "isValidRelatedTable calls materializeLatest() bypassing any pin; comment at :783-786 acknowledges the remote enumeration cost but assumes 'once per refresh'." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 114, + "note": "getOrMaterialize: empty Optional -> materializeLatest(); getNameToPartitionItems:580-582, getAndCopyPartitionItems:585-587, getPartitionType:590-591, getPartitionColumns:603-604 all route through it." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescInitGenerator.java", + "line": 43, + "note": "getAndCopyPartitionItems(MvccUtil.getSnapshotFromContext(relatedTable)) per pct table; runs once per generateRelatedPartitionDescs pass (twice per refresh: alignMvPartition and calculatePartitionMappings)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java", + "line": 138, + "note": "alignMvPartition -> generateRelatedPartitionDescs (:187-193 iterates partitionDescGenerators)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java", + "line": 539, + "note": "calculatePartitionMappings -> generateRelatedPartitionDescs AGAIN, invoked from MTMVRefreshContext.buildContext (MTMVRefreshContext.java:63)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshContext.java", + "line": 35, + "note": "baseTableSnapshotCache caches only MTMVSnapshotIf per table — no refresh-scoped partition-view/MvccSnapshot pin exists." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java", + "line": 1015, + "note": "getSnapshot(TableIf) is a passive snapshots-map lookup (no lazy load); map is populated only by statement planning, so it is empty at the MTMVTask call sites -> every accessor re-materializes." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescRollUpGenerator.java", + "line": 61, + "note": "EXPR-only: getPartitionType(empty) -> one more materializeLatest per pass; rollUpRange :145-147 then calls generateRollUpPartitionKeyDesc PER partition desc." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java", + "line": 626, + "note": "getPartitionColumnType -> relatedTable.getPartitionColumns(getSnapshotFromContext=empty) -> getOrMaterialize -> materializeLatest; called per partition desc from MTMVPartitionExprDateTrunc.java:134-135 inside the rollUpRange loop (worse than the claimed '+1'). Note: DateTrunc:77 cited in the claim is create-time analyze(), not the refresh path." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1453, + "note": "getMvccPartitionView: uncached catalogOps.loadTable + buildMvccPartitionView per call; getTableHandle:324 adds a remote tableExists RPC per materializeLatest; beginQuerySnapshot:1540 TTL-caches only the (snapshotId,schemaId) pin, not the enumeration." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 712, + "note": "partitionsTable.newScan().useSnapshot(snapshotId).planFiles() — PARTITIONS metadata-table scan = remote manifest-list+manifest IO per the SDK cost model; comment at :526 states it must run in auth context because it is a remote read." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "loadTable is a raw catalog.loadTable SDK delegation — no CachingCatalog wrap found in fe-connector-iceberg or fe-connector-metastore-iceberg." + } + ], + "corrected_multiplicity": "Per refresh per external (iceberg/paimon MVCC) pct table: minimum k=3 full materializeLatest rounds for non-EXPR MTMVs (isValidRelatedTable + InitGenerator in alignMvPartition + InitGenerator in buildContext/calculatePartitionMappings). For EXPR (date_trunc) MTMVs: k = 5 + 2P, where P = number of related partition descs — the RollUp getPartitionType (x2 passes) plus getPartitionColumnType PER partition desc inside rollUpRange (x2 passes) each independently re-materialize. The claimed 4-6 is correct in order for small tables but UNDERSTATES the EXPR case, which degrades to per-partition multiplicity. All k rounds occur before the NOT_REFRESH short-circuit, so even a no-op refresh pays them.", + "mitigation_found": "Partial only, none memoize the partition view: (1) IcebergLatestSnapshotCache (IcebergConnectorMetadata.java:1540) TTL-caches only the (snapshotId, schemaId) pin used by beginQuerySnapshot — the PARTITIONS enumeration in getMvccPartitionView is never cached; (2) the SDK manifest content cache (io.manifest.cache-enabled derived to \"true\", IcebergCatalogFactory.java:131) can serve repeated manifest bytes from memory across the k PARTITIONS scans within one refresh, reducing remote bytes but not the per-call tableExists RPC, loadTable metadata read (no CachingCatalog), manifest-list read, or Avro decode/plan CPU; (3) MTMVRefreshContext.baseTableSnapshotCache (MTMVRefreshContext.java:35) caches MTMVSnapshotIf table snapshots only.", + "fix_direction": "Introduce a refresh-scoped MvccSnapshot pin: materialize once at the top of MTMVTask.run (or lazily in MTMVRefreshContext with a per-table Map<BaseTableInfo, MvccSnapshot> alongside the existing baseTableSnapshotCache) and thread the Optional<MvccSnapshot> through isValidRelatedTable / generateRelatedPartitionDescs / calculatePartitionMappings / getPartitionColumnType, so all k enumeration points read one pinned PluginDrivenMvccSnapshot. This also closes the snapshot-skew hazard between the k independent enumerations. Cheaper interim: a per-refresh memo inside PluginDrivenMvccExternalTable keyed by refresh context, or a cheap specs-only eligibility SPI for isValidRelatedTable (already suggested in its own comment at PluginDrivenMvccExternalTable.java:785-786).", + "impact_estimate": "For a non-EXPR MTMV over one iceberg pct table: 3x (tableExists RPC + uncached loadTable + PARTITIONS planFiles) per refresh where 1x would do — roughly 2 redundant rounds, each hundreds of ms to seconds on manifest-heavy tables (manifest content cache may soften repeats within the same JVM). For a date_trunc EXPR MTMV with P partitions: 5+2P rounds — e.g. P=500 partitions => ~1000 redundant loadTable+PARTITIONS scans per refresh, easily minutes of pure metadata IO, on every scheduled refresh including no-ops. Independent of and additive to the per-partition loop finding (fixing that does not remove these pre-loop duplicates)." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "Every hop the finder cited is real and un-memoized. During MTMVTask.run the ConnectContext has no StatementContext snapshot pin: MTMVPlanUtil.getBaseTableFromQuery plans the MV query inside a TEMP StatementContext and restores the original (null) in finally (MTMVPlanUtil.java:221-230), so every later MvccUtil.getSnapshotFromContext returns empty and PluginDrivenMvccExternalTable.getOrMaterialize(empty) (lines 114-119, no cached field) runs materializeLatest() from scratch each time. Each materializeLatest = remote tableExists RPC (IcebergConnectorMetadata.getTableHandle:318-332) + beginQuerySnapshot + getMvccPartitionView, where getMvccPartitionView does an UNCACHED catalogOps.loadTable (:1453; raw SDK catalog.loadTable at IcebergCatalogOps.java:340-341, no CachingCatalog wrap) + a PARTITIONS metadata-table newScan().useSnapshot(id).planFiles() (IcebergPartitionUtils.java:709-723) reading manifest-list + all manifests remotely. Mitigations found are all partial or off-by-default: (a) IcebergLatestSnapshotCache (default TTL 86400s, IcebergConnector.java:130) dedups only the beginQuerySnapshot sub-step; (b) MTMVRefreshContext.baseTableSnapshotCache (:35) caches only table-level MTMVSnapshotIf, never the partition view, and does not exist during hops 1-2; (c) the SDK manifest cache io.manifest.cache-enabled is default-disabled (IcebergCatalogFactory.java:68 DEFAULT_MANIFEST_CACHE_ENABLE=false); (d) Doris's IcebergManifestCache serves the data-scan planFiles path (IcebergScanPlanProvider), not the PARTITIONS metadata-table scan. Moreover the finder UNDERCOUNTED: for EXPR/date_trunc MVs, rollUpRange (MTMVRelatedPartitionDescRollUpGenerator.java:145-147) calls generateRollUpPartitionKeyDesc PER PARTITION DESC, each hitting MTMVPartitionUtil.getPartitionColumnType (:626) -> getPartitionColumns(empty) -> unconditional getOrMaterialize (PluginDrivenMvccExternalTable.java:603-604), turning the pre-loop redundancy into O(partitions) full remote materializations per generateRelatedPartitionDescs pass — which itself runs twice (alignMvPartition at MTMVPartitionUtil.java:136-139 and calculatePartitionMappings at MTMV.java:539-540). isValidRelatedTable (:787) adds one more, its own comment ('Bounded — once per refresh — acceptable') showing the per-call cost was accepted without seeing the aggregate flow. Master's IcebergSnapshotCacheValue-style snapshot cache that used to absorb these repeats has no equivalent in the SPI branch.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 114, + "note": "getOrMaterialize: empty Optional -> materializeLatest() every call; no cached field, no memo (lines 114-119, 126-193)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 787, + "note": "isValidRelatedTable always materializeLatest (bypasses pin); comment at 782-786 accepts cost as 'once per refresh' without the aggregate view." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 603, + "note": "getPartitionColumns(snapshot) unconditionally calls getOrMaterialize FIRST — every empty-snapshot call is a full remote materialization." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java", + "line": 221, + "note": "Hop 1 isValidRelatedTable per pct table; hop 2 alignMvPartition at :228; hop 3 MTMVRefreshContext.buildContext at :243 — all in one refresh flow." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java", + "line": 230, + "note": "getBaseTableFromQuery restores the original (null) StatementContext in finally — snapshot pins from the planning pass are DISCARDED, so getSnapshotFromContext is empty for all later hops." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescInitGenerator.java", + "line": 43, + "note": "getAndCopyPartitionItems(MvccUtil.getSnapshotFromContext(...)) -> empty -> materializeLatest; runs once per generateRelatedPartitionDescs pass (twice per refresh: MTMVPartitionUtil.java:138 and MTMV.java:539)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedPartitionDescRollUpGenerator.java", + "line": 145, + "note": "UNDERCOUNTED HOP: rollUpRange loops every partition desc -> generateRollUpPartitionKeyDesc -> MTMVPartitionExprDateTrunc.java:134 -> MTMVPartitionUtil.getPartitionColumnType(:626) -> getPartitionColumns(empty) -> one FULL materializeLatest PER PARTITION (plus getPartitionType at :61)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshContext.java", + "line": 35, + "note": "Only mitigation at context level: baseTableSnapshotCache caches table-level MTMVSnapshotIf — NOT the partition view; also not yet constructed during hops 1-2." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1453, + "note": "getMvccPartitionView: fresh catalogOps.loadTable (uncached — raw SDK catalog.loadTable, IcebergCatalogOps.java:340-341, no CachingCatalog) + buildMvccPartitionView per call." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1540, + "note": "PARTIAL mitigation: beginQuerySnapshot served by IcebergLatestSnapshotCache (default TTL 86400s, IcebergConnector.java:130) — dedups only this sub-step's loadTable, not the partition-view loadTable/scan; tableExists at :324 also remote per materialize." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 712, + "note": "loadRawPartitions: PARTITIONS metadata-table newScan().useSnapshot(id).planFiles() — remote manifest-list + manifest reads on every materializeLatest; not routed through Doris IcebergManifestCache (data-scan only)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "SDK-level manifest cache (io.manifest.cache-enabled) default-DISABLED (DEFAULT_MANIFEST_CACHE_ENABLE = false) — the only layer that could soften repeated PARTITIONS scans is off by default." + } + ], + "mitigation_found": "Partial only. (1) IcebergLatestSnapshotCache (default TTL 86400s) dedups the beginQuerySnapshot loadTable inside each materializeLatest — but not the getMvccPartitionView loadTable, the tableExists RPC, or the PARTITIONS planFiles. (2) MTMVRefreshContext.baseTableSnapshotCache caches only table-level MTMVSnapshotIf per context, never the partition view, and doesn't exist during hops 1-2. (3) SDK io.manifest.cache-enabled is default-disabled (IcebergCatalogFactory.java:68). (4) Doris IcebergManifestCache serves only the data-scan planFiles path. (5) No CachingCatalog wrap around catalogOps. (6) StatementContext snapshot pins from the MV-query planning pass are discarded (MTMVPlanUtil.java:230) before any of the cited hops run. Nothing prevents the repeated tableExists + loadTable + PARTITIONS scans in the default configuration.", + "corrected_multiplicity": "Finder's k~4-6 is correct for the base flow but UNDERCOUNTS the EXPR case. Non-EXPR (FOLLOW_BASE_TABLE) MV: k=3 full materializations per refresh per external pct table (isValidRelatedTable + alignMvPartition/InitGenerator + calculatePartitionMappings/InitGenerator). EXPR(date_trunc) MV: each of the two generateRelatedPartitionDescs passes costs 2+P materializations (InitGenerator + RollUp getPartitionType + P per-partition getPartitionColumnType via rollUpRange -> generateRollUpPartitionKeyDesc), so ~2*(2+P)+1 = O(2P+5) where P = base-table partition count — per-partition multiplicity, not a constant k.", + "impact_estimate": "Multiplicity: O(k=3) per refresh per pct table for FOLLOW_BASE_TABLE MVs; O(partitions) (~2P+5) for EXPR/date_trunc MVs. Cost class: remote-IO per materialization — 1 metastore tableExists RPC + 1 uncached catalog.loadTable (metadata JSON fetch) + 1 PARTITIONS metadata-table planFiles (manifest-list + all manifests via FileIO; SDK manifest cache off by default) — hundreds of ms to seconds each on manifest-heavy tables. Trigger breadth: every scheduled or manual refresh of any MTMV whose pct/base table is a plugin-MVCC external table (iceberg range-eligible tables; paimon via the same generic path), including no-op refreshes that end in NOT_REFRESH. Only the beginQuerySnapshot sub-step is deduped by the default-on latest-snapshot cache. An EXPR MV over a 1000-partition iceberg table pays ~2005 remote materialization rounds per refresh where master paid ~1 (its IcebergSnapshotCacheValue absorbed repeats).", + "fix_direction": "Memoize + pass-down: add a refresh-flow-scoped pin of the materialized PluginDrivenMvccSnapshot — e.g. extend MTMVRefreshContext (or pin in the task's StatementContext before hop 1) to hold the MvccSnapshot per pct table and thread it through the Optional parameters every accessor already accepts; alternatively memoize materializeLatest keyed by the (already-cached) latest snapshot id. Independently, hoist the loop-invariant getPartitionColumnType out of rollUpRange's per-partition loop (compute once per pct table). A cheap specs-only eligibility SPI for isValidRelatedTable (already noted as future work in its own comment) removes hop 1's full partition enumeration." + } + }, + { + "title": "Every query on a partitioned iceberg table pays a PARTITIONS metadata-table scan at plan time with no cross-query cache (legacy second-level cache deliberately dropped)", + "variant": "C-cache-bypass", + "heavy_op": "materializeLatest during BindRelation snapshot pinning: catalogOps.loadTable + PARTITIONS metadata table planFiles (buildMvccPartitionView at IcebergPartitionUtils.java:565->712 for time-transform tables, or listLatestPartitions -> IcebergConnectorMetadata.listPartitions -> loadRawPartitions planFiles at IcebergConnectorMetadata.java:1500-1519 / IcebergPartitionUtils.java:641->712 for identity/bucket-partitioned tables)", + "multiplicity": "per-query (multiplier = QPS x table references): BindRelation.getLogicalPlan runs loadSnapshots for every Mvcc table reference in every statement (BindRelation.java:733); within one statement StatementContext.snapshots dedupes (StatementContext.java:988-998), across statements nothing does", + "entry_chain": "BindRelation.getLogicalPlan (BindRelation.java:733) -> StatementContext.loadSnapshots (StatementContext.java:988-998) -> PluginDrivenMvccExternalTable.loadSnapshot (PluginDrivenMvccExternalTable.java:343-346) -> materializeLatest (PluginDrivenMvccExternalTable.java:126-193; getMvccPartitionView at :165, listLatestPartitions at :181/:190 -> metadata.listPartitions at :270) -> IcebergConnectorMetadata.getMvccPartitionView/listPartitions (IcebergConnectorMetadata.java:1453/1507 loadTable) -> IcebergPartitionUtils.buildMvccPartitionView/listPartitions (IcebergPartitionUtils.java:565/641) -> loadRawPartitions planFiles (IcebergPartitionUtils.java:709-712)", + "files": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java", + "line": 733 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 181 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1507 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 641 + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java", + "line": 826 + } + ], + "why_heavy": "The PARTITIONS metadata scan reads the manifest list + all manifests (O(manifests) remote IO + avro decode) and is paid ON TOP of the query's own data planFiles — i.e., manifest metadata is traversed twice per query. Its result (partition set at snapshot S) is a pure function of the snapshot id, which the connector already has (beginQuerySnapshot is TTL-cached in IcebergLatestSnapshotCache, IcebergConnectorMetadata.java:1536-1548), making it trivially memoizable keyed by (table, snapshotId). Legacy master cached exactly this in IcebergMetadataCache; the cutover explicitly removed the second level (comment 'CACHE-P1: the cutover lists partitions per query instead of maintaining a second-level cache', PluginDrivenExternalTable.java:826-827) — the only remaining cache (IcebergLatestSnapshotCache) covers the snapshot-id pin, not the partition enumeration, so the hot path bypasses any cache every query.", + "est_impact": "For a partitioned iceberg table with M manifests: +1 loadTable RPC + 1 manifest-list read + up to M manifest reads/decodes per query at plan time (partially softened by the iceberg FileIO manifest byte cache when derived on). At high QPS this roughly doubles per-query metadata IO vs the data-planning-only cost and vs legacy cache-hit behavior; used only to feed selectedPartitionNum/pruning display and the pin's partition map.", + "gate": "Partitioned iceberg tables only (unpartitioned early-returns after the loadTable at IcebergPartitionUtils.java:632-637). Deliberate CACHE-P1 design decision — reported because the heavy op sits at per-query multiplicity with a snapshot-id key readily available for memoization.", + "confidence": "medium", + "lens": "stats-partitions-freshness", + "dupes": [], + "refute": { + "verdict": "CONFIRMED", + "reasoning": "Independently re-traced the full chain from the actual code. (1) Heaviness holds: IcebergPartitionUtils.loadRawPartitions (709-712) runs partitionsTable.newScan().useSnapshot(id).planFiles() + row materialization — a PARTITIONS metadata-table scan, explicitly remote IO (manifest-list + all manifests + avro decode) per the SDK cost model — and is fronted by catalogOps.loadTable (IcebergConnectorMetadata 1453/1507), a thin delegation to catalog.loadTable (IcebergCatalogOps 340-342) with no CachingCatalog wrap anywhere in the connector. (2) Multiplicity holds: BindRelation.getLogicalPlan:733 calls StatementContext.loadSnapshots for every Mvcc table reference; the dedup map (StatementContext 993-996) is an instance field of the per-statement StatementContext, so across statements nothing dedupes; PluginDrivenMvccExternalTable.loadSnapshot(empty,empty) (343-346) -> materializeLatest (126-193) has zero caching, and IcebergConnector.getMetadata (212-215) builds a fresh IcebergConnectorMetadata per call. (3) No mitigating cache on this path: IcebergLatestSnapshotCache stores only (snapshotId, schemaId) for beginQuerySnapshot (IcebergConnectorMetadata 1536-1548; value shape IcebergLatestSnapshotCache 57-63) — it pins the snapshot id but not the partition view; the connector-owned IcebergManifestCache is consumed only by IcebergScanPlanProvider (data-scan planning), never by getMvccPartitionView/listPartitions/loadRawPartitions; the only softening is the SDK FileIO manifest byte cache (io.manifest.cache-enabled derived true, IcebergCatalogFactory 114-131), which the candidate already discounted — loadTable RPC and avro decode remain per query. (4) Reachability holds: IcebergConnector declares SUPPORTS_MVCC_SNAPSHOT (652) so iceberg tables build as PluginDrivenMvccExternalTable (PluginDrivenExternalDatabase 57-61). The deliberate no-second-level-cache design is documented at PluginDrivenExternalTable 826-827 (CACHE-P1). The cost is AMPLIFIED, not inherent: the partition view is a pure function of (table, snapshotId), the snapshot id is already TTL-cached and stable within the TTL, and the query's own data planFiles independently traverses the same manifests — so the same manifest metadata is read twice per query and recomputed identically across queries at the same snapshot. Sub-case is worse than claimed: for identity/bucket-partitioned tables materializeLatest triggers TWO remote loadTable calls per query (getMvccPartitionView at 1453 loads, buildMvccPartitionView early-returns UNPARTITIONED at IcebergPartitionUtils 533-534, then listLatestPartitions -> listPartitions loads AGAIN at 1507 before the scan at 641->709-712). Gate as stated: unpartitioned tables skip the PARTITIONS scan (empty spec -> isValidRelatedTable false -> unpartitioned view; empty partition columns skip listLatestPartitions) but still pay one loadTable per query via the unconditional getMvccPartitionView call at materializeLatest:165.", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java", + "line": 733, + "note": "getLogicalPlan calls cascadesContext.getStatementContext().loadSnapshots(table, ...) unconditionally for every bound relation before the table-type switch — per table reference per statement." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java", + "line": 993, + "note": "loadSnapshots dedupes via the per-StatementContext instance map 'snapshots' (declared line 272); dedup scope is one statement only — a new StatementContext per statement means loadSnapshot re-runs every query." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 343, + "note": "loadSnapshot with no time-travel spec (the normal query path) calls materializeLatest() at 346; materializeLatest (126-193) has no cache: beginQuerySnapshot at 156, getMvccPartitionView at 165, listLatestPartitions at 181 (non-RANGE view + partition columns) and 190 (no view), which calls metadata.listPartitions at 270." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1453, + "note": "getMvccPartitionView: catalogOps.loadTable (remote RPC) then IcebergPartitionUtils.buildMvccPartitionView at 1454; no cache. listPartitions does a second loadTable at 1507 then IcebergPartitionUtils.listPartitions at 1513. beginQuerySnapshot (1536-1548) is the ONLY cached call: latestSnapshotCache.getOrLoad at 1540 returns just (snapshotId, schemaId)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 712, + "note": "loadRawPartitions (709-723): MetadataTableUtils.createMetadataTableInstance(table, PARTITIONS) then partitionsTable.newScan().useSnapshot(snapshotId).planFiles() + task.asDataTask().rows() — the PARTITIONS metadata-table scan, remote IO per the cost model. Reached from buildMvccPartitionView:565 (time-transform tables, after eligibility gate 533-534) and listPartitions:641 (any partitioned table; unpartitioned/empty early-return 632-637)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "CatalogBackedIcebergCatalogOps.loadTable is a thin delegation to catalog.loadTable(toTableIdentifier(...)) — no CachingCatalog wrap (grep for CachingCatalog in the connector finds none), so each call is a live metastore/REST round-trip." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 212, + "note": "getMetadata builds a NEW IcebergConnectorMetadata per call (213-215); the only shared state is latestSnapshotCache (default TTL 86400s, line 130) which caches the pin ids, not the partition view. SUPPORTS_MVCC_SNAPSHOT declared at 652 makes iceberg tables PluginDrivenMvccExternalTable (PluginDrivenExternalDatabase.java 57-61)." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java", + "line": 826, + "note": "Comment confirms the deliberate design: 'One round-trip, no FE-side partition-value cache (per CACHE-P1: the cutover lists partitions per query instead of maintaining a second-level cache)'. Legacy master's IcebergMetadataCache second level was dropped." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java", + "line": 218, + "note": "IcebergManifestCache is wired only into the scan-plan provider (data planning path); grep shows no usage in IcebergConnectorMetadata/IcebergPartitionUtils — the PARTITIONS scan bypasses it entirely." + } + ], + "corrected_multiplicity": "per-query as claimed (once per distinct iceberg Mvcc table reference per statement; within-statement dedup only, no cross-statement memoization). Worse than claimed for identity/bucket-partitioned tables: 2 remote loadTable RPCs + 1 PARTITIONS metadata scan per query (loadTable at IcebergConnectorMetadata:1453 for the view probe, again at :1507 for the LIST fallback). Time-transform tables: 1 loadTable + 1 PARTITIONS scan. Unpartitioned tables: 1 loadTable per query (getMvccPartitionView probe), no PARTITIONS scan.", + "mitigation_found": "Partial only, and none on the partition-enumeration itself: (1) IcebergLatestSnapshotCache (TTL default 86400s) caches only the (snapshotId, schemaId) pin for beginQuerySnapshot — not the partition view; (2) the SDK FileIO manifest byte cache (io.manifest.cache-enabled derived true by default, IcebergCatalogFactory.java:114-131) softens repeated manifest byte reads but not the loadTable RPC or the per-query avro decode/row materialization; (3) IcebergManifestCache covers only the data-scan planning path. StatementContext.snapshots dedupes within one statement only. CACHE-P1 comment documents the missing cross-query cache as a deliberate cutover decision.", + "fix_direction": "Memoize the ConnectorMvccPartitionView / listPartitions result keyed by (table identity, snapshotId) in a connector-owned TTL cache alongside IcebergLatestSnapshotCache (or via the fe-connector-cache framework) — the result is a pure function of the snapshot id the connector already pins, and cache coherence follows for free from the existing snapshot-pin TTL. Secondary: within one materializeLatest, thread the already-loaded Table (or at least reuse one loadTable) across the getMvccPartitionView probe and the listPartitions fallback to eliminate the duplicate loadTable RPC for identity/bucket-partitioned tables.", + "impact_estimate": "Every statement referencing a partitioned iceberg table pays, at plan time and on top of the data scan's own manifest traversal: 1-2 catalog.loadTable RPCs + 1 PARTITIONS metadata scan (manifest-list read + up to M manifest reads/decodes; byte reads partially softened by the FileIO manifest cache, decode+row materialization not). At steady QPS against a table with hundreds+ manifests this roughly doubles per-query metadata work vs data-planning-only and regresses vs legacy IcebergMetadataCache cache-hit behavior; the output feeds only selectedPartitionNum/EXPLAIN partition display, SQL-block-rule enforcement, and the pin's partition map." + }, + "mitigation": { + "verdict": "CONFIRMED", + "reasoning": "The candidate's chain is real and no layer memoizes the heavy op in default configuration. (1) Entry: BindRelation.getLogicalPlan calls StatementContext.loadSnapshots for every relation (BindRelation.java:733); the snapshots map dedupes only WITHIN one statement (StatementContext.java:993-996, map lives on the per-statement context) — nothing dedupes across statements. (2) PluginDrivenMvccExternalTable.loadSnapshot -> materializeLatest runs per statement (PluginDrivenMvccExternalTable.java:343-346, 126-193); it always calls metadata.getMvccPartitionView (:165), and for a non-RANGE view on a table with partition columns additionally listLatestPartitions -> metadata.listPartitions (:181, :270). (3) Both connector methods do a BARE catalogOps.loadTable (IcebergConnectorMetadata.java:1453 and :1507) — IcebergCatalogOps.CatalogBacked.loadTable is a direct catalog.loadTable with no CachingCatalog anywhere in the connector/metastore modules (IcebergCatalogOps.java:340-341; grep for CachingCatalog: zero hits) — then scan the PARTITIONS metadata table: buildMvccPartitionView -> loadRawPartitions (IcebergPartitionUtils.java:565) or listPartitions -> loadRawPartitions (:641), which is partitionsTable.newScan().useSnapshot(id).planFiles() + row iteration (:712-717) = manifest-list read + O(manifests) manifest reads/avro decode, remote IO. Mitigation hunt results, all partial or off-by-default: (a) IcebergLatestSnapshotCache (default TTL 86400s, IcebergConnector.java:130,188-189) caches ONLY the beginQuerySnapshot (snapshotId, schemaId) pin (IcebergConnectorMetadata.java:1540-1545) — the partition enumeration and its loadTables run outside it; (b) the iceberg FileIO manifest byte cache is derived on only from meta.cache.iceberg.manifest.enable, whose default is FALSE (DEFAULT_MANIFEST_CACHE_ENABLE=false, IcebergCatalogFactory.java:68, derivation :120-133), so in default config every PARTITIONS scan re-reads all manifests remotely; (c) IcebergManifestCache is also gated on that same default-off knob and is consumed only by the data-scan provider — IcebergPartitionUtils has zero references to it (IcebergConnector.java:159-160); (d) the FE second-level partition cache was deliberately dropped (CACHE-P1 comment, PluginDrivenExternalTable.java:826-827); (e) downstream readers (getNameToPartitionItems/getPartitionType at PluginDrivenMvccExternalTable.java:580-599) correctly reuse the statement pin, so there is no intra-statement amplification — the problem is purely cross-statement. Iceberg declares SUPPORTS_MVCC_SNAPSHOT (IcebergConnector.java:652) so every iceberg table reference takes this path. Legacy master served this from IcebergMetadataCache within TTL; the cutover replaced a cache hit with a full remote enumeration per query, keyed data (pure function of snapshotId, which the TTL-cached pin already supplies) making it trivially memoizable. One refinement: for identity/bucket/truncate-partitioned tables the path pays TWO uncached loadTables per statement (getMvccPartitionView loads, returns unpartitioned verdict without scanning; listPartitions loads again before the single PARTITIONS scan), and even unpartitioned iceberg tables pay one uncached loadTable per query (early return at IcebergPartitionUtils.java:632-637 / isValidRelatedTable=false comes after the load at IcebergConnectorMetadata.java:1453).", + "evidence": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java", + "line": 733, + "note": "loadSnapshots invoked for every table reference during binding (before the type switch), so every statement referencing an Mvcc table triggers a pin materialization." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java", + "line": 993, + "note": "snapshots.containsKey dedupe is per-StatementContext (per statement, keyed by table+version selector); no cross-statement layer exists here. Note: file is nereids/StatementContext.java, not qe/." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 165, + "note": "materializeLatest unconditionally calls metadata.getMvccPartitionView; non-RANGE+partitioned falls through to listLatestPartitions (:181) -> metadata.listPartitions (:270). loadSnapshot(:343-346) returns materializeLatest() with no field caching." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java", + "line": 1453, + "note": "getMvccPartitionView: bare catalogOps.loadTable + buildMvccPartitionView per call; listPartitions repeats loadTable at :1507. Only beginQuerySnapshot (:1540) goes through IcebergLatestSnapshotCache." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 712, + "note": "loadRawPartitions: PARTITIONS metadata table newScan().useSnapshot(id).planFiles() + row iteration = manifest-list + O(manifests) remote reads; called from buildMvccPartitionView (:565) and listPartitions (:641); no cache consulted (zero IcebergManifestCache references in this file)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogOps.java", + "line": 340, + "note": "CatalogBacked.loadTable = catalog.loadTable(toTableIdentifier(...)) directly; no CachingCatalog wrap anywhere in fe-connector-iceberg or fe-connector-metastore-iceberg (grep zero hits)." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java", + "line": 68, + "note": "DEFAULT_MANIFEST_CACHE_ENABLE = false: io.manifest.cache-enabled is derived true only when the user sets meta.cache.iceberg.manifest.* or the raw iceberg key (:120-133) — the FileIO manifest byte cache is OFF by default, so the candidate's 'partially softened' caveat does not apply in default config." + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java", + "line": 159, + "note": "Comment confirms manifestCache is consumed only when meta.cache.iceberg.manifest.enable is set (default off); latestSnapshotCache built with DEFAULT_TABLE_CACHE_TTL_SECOND=86400 (:130,:188-189) covers only the snapshot pin; SUPPORTS_MVCC_SNAPSHOT declared at :652 so iceberg tables live on the Mvcc path." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java", + "line": 826, + "note": "CACHE-P1 comment: 'the cutover lists partitions per query instead of maintaining a second-level cache' — explicit confirmation the legacy cross-query partition cache was dropped by design." + }, + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 580, + "note": "Downstream readers (getNameToPartitionItems/getPartitionType) read the statement pin via getOrMaterialize(snapshot) — no intra-statement re-listing; the amplification is strictly cross-statement (per query)." + } + ], + "mitigation_found": "Four partial/off-by-default mechanisms, none covering the heavy op in default config: (1) IcebergLatestSnapshotCache (TTL default 24h) caches only the beginQuerySnapshot (snapshotId, schemaId) pin — the partition enumeration's own loadTable + PARTITIONS planFiles bypass it entirely; (2) StatementContext.snapshots dedupes only within one statement; (3) the iceberg FileIO manifest byte cache (io.manifest.cache-enabled) is default-OFF (derived from meta.cache.iceberg.manifest.enable, DEFAULT_MANIFEST_CACHE_ENABLE=false) and even when on only avoids manifest byte re-fetch, not the loadTable RPC, manifest-list read, or avro decode; (4) IcebergManifestCache is gated on the same default-off knob and serves only the data-scan provider, never loadRawPartitions. No CachingCatalog wrap on catalogOps.loadTable. CACHE-P1 comment confirms the legacy cross-query second-level cache was deliberately dropped.", + "corrected_multiplicity": "Per statement per distinct (iceberg table, version-selector) reference — dedupe exists within one statement (StatementContext.snapshots) but nothing across statements, so effective rate = QPS x table references. Per-statement cost on the default config: time-transform-partitioned table = 1 uncached loadTable + 1 PARTITIONS metadata-table planFiles (all manifests, remote); identity/bucket/truncate-partitioned = 2 uncached loadTables (getMvccPartitionView loads then verdicts UNPARTITIONED without scanning; listPartitions loads AGAIN) + 1 PARTITIONS planFiles; unpartitioned = 1 uncached loadTable, no PARTITIONS scan. The beginQuerySnapshot loadTable is the only one saved (TTL cache).", + "impact_estimate": "O(1 per table reference per query, i.e. multiplier = QPS x references) x remote-IO (1-2 catalog.loadTable RPC + metadata-JSON reads, plus manifest-list read + O(M manifests) manifest fetches/avro decodes for partitioned tables) x broad trigger: EVERY query that binds an iceberg table through the plugin path pays at least the uncached loadTable; every PARTITIONED iceberg table additionally pays the full PARTITIONS metadata scan at bind time, on top of the query's own inherent data planFiles — roughly doubling plan-time metadata IO per query and regressing vs legacy master, which served this from IcebergMetadataCache within TTL. Default-off manifest byte cache means no softening in default deployments.", + "fix_direction": "Memoize keyed by (TableIdentifier, snapshotId): the pin from TTL-cached beginQuerySnapshot is threaded onto the handle (applySnapshot) BEFORE getMvccPartitionView/listPartitions run, so a per-catalog fe-connector-cache MetaCacheEntry (mirroring IcebergLatestSnapshotCache, invalidated together on REFRESH TABLE/CATALOG) holding the ConnectorMvccPartitionView / List collapses repeat queries to in-memory hits — the result is a pure function of the snapshot id, restoring legacy IcebergMetadataCache semantics inside the connector (consistent with the connector-owned-caching architecture). Secondary hoist: the identity/bucket path loads the table twice per statement (IcebergConnectorMetadata:1453 then :1507); pass the loaded Table (or fold the LIST fallback into getMvccPartitionView) to drop one loadTable RPC." + } + } + ], + "refuted": [ + { + "title": "Dictionary freshness poll (getNewestUpdateVersionOrTime) re-materializes the full partition view on every poll, bypassing any pin by design", + "lens": "stats-partitions-freshness", + "files": [ + { + "file": "fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java", + "line": 748 + }, + { + "file": "fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java", + "line": 712 + } + ], + "refuteReason": "The heavy op is real exactly as claimed: PluginDrivenMvccExternalTable.getNewestUpdateVersionOrTime always calls materializeLatest() (\"always probe LATEST (bypass any context pin)\"), which for iceberg does an uncached remote catalogOps.loadTable plus a PARTITIONS metadata-table planFiles (manifest-list + manifests) for partitioned tables, with no memoization at any layer — and since beginQuerySnapshot is served by the 24h-TTL IcebergLatestSnapshotCache, the enumeration snapshot is pinned constant within the TTL, so every repeat is deterministically redundant. HOWEVER the claimed multiplicity driver collapses: the dictionary poll (DictionaryManager, 5s default) can never reach this method because a dictionary over an external-catalog table cannot be created — CreateDictionaryInfo.validateAndSet line 164 casts the source table to org.apache.doris.catalog.Table, and ExternalTable implements only TableIf (does not extend Table), so CREATE DICTIONARY over any external source throws ClassCastException during validation, before DictionaryManager.createDictionary registers anything. createDictionary has no other caller, the file is untouched by this branch (upstream-only history, so no legacy image can carry such a dictionary), and no regression test creates an external-source dictionary. The gate is not rare — it is unreachable, so the poll-path cost is never incurred. Per the refutation rules (multiplicity collapses), the finding as scoped is REFUTED. Side discovery for a separate finding: the SAME heavy method IS reachable at per-query multiplicity via the SQL-cache validity probe (NereidsSqlCacheManager.java:476-481 explicitly for \"flipped hive/iceberg/paimon/hudi\" external MVCC tables, plus SqlCacheContext.java:200 and CacheAnalyzer.java:489 at cache-entry build) when enable_sql_cache is on — a different entry chain and gate than the candidate's.", + "mitigation": "Partial/off-path only. (1) IcebergLatestSnapshotCache (default on, TTL 86400s access-based) covers ONLY the beginQuerySnapshot pin — and by keeping the pinned snapshot id constant it makes every poll's PARTITIONS scan provably redundant rather than preventing it. (2) Doris IcebergManifestCache: wired only into IcebergScanPlanProvider, not this path. (3) SDK io.manifest.cache-enabled: default-disabled (DEFAULT_MANIFEST_CACHE_ENABLE=false); user opt-in would absorb only repeated manifest byte reads. (4) Cache-backed getTableFreshness short-circuit exists but is deliberately hive(last-modified)-only. (5) No table-object cache/CachingCatalog anywhere; no memo at the Dictionary layer." + } + ] +} \ No newline at end of file diff --git a/plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17.md b/plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17.md new file mode 100644 index 00000000000000..3ff339476a7d40 --- /dev/null +++ b/plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17.md @@ -0,0 +1,220 @@ +# fe-connector-iceberg 热路径重操作审计报告 + +日期：2026-07-17 +问题类定义：见 `plan-doc/perf-heavy-op-hot-path-problem-class.md`（由 DORIS-27138 问题一 / PR #64134 泛化） +完整证据（全部调用链 + 两路验证意见）：`plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17-findings.json` + +## 0. 方法与可信度 + +- 多 agent 对抗审计（55 agent）：7 个视角的 finder 并行扫描（per-split 循环放大 / 单链路重复 / + 隐藏重访问器 / per-split 序列化 / 缓存旁路 / 写提交路径 / stats-分区-MTMV 路径） + → 33 条原始发现 → 去重 24 条 → 每条经 2 个独立对抗 verifier + （一个专职反驳乘数与成本、一个专职猎找既有缓解机制）→ **23 条确认、1 条驳回、0 存疑**。 +- 人工抽验：C1/C2/C6 的核心代码主张（`getColumnHandles:587` 远程 loadTable、 + `getScanNodeProperties:1300/1311` 二次 loadTable + planFiles 兜底、`IcebergCatalogOps:340` 裸委派、 + 全仓 grep 无 CachingCatalog、`getOrLoadPropertiesResult` memoize + `convertPredicate:796` 失效点） + 均已由我逐行核对属实。 +- 行号基于本分支 `catalog-spi-review-16` 当前工作树。 + +## 1. 结论速览（按严重度分层） + +**总病灶一句话：新框架的 SPI 各入口"每次自己 loadTable / 自己扫"，而 legacy 有单一 source +持缓存 Table —— 同一信息在一次规划里被远程重取 3~7 次；另有 #64134 的 planFiles 兜底 +在新框架以 per-query 乘数复活。** + +| 级别 | 簇 | 一句话 | 乘数 × 成本 | 触发面 | 发现项 | +|---|---|---|---|---|---| +| **P0** | 簇1 无 Table 对象缓存 | 一次 SELECT 规划 **3~7 次远程 loadTable**（同一张表） | k≈3-7/查询 × metastore RPC + metadata.json | 所有 iceberg 查询 | C1 C4 C6 C10 C16 | +| **P0** | 簇3 分区视图无跨查询缓存 | 分区表**每个查询**在分析期做一次 PARTITIONS 元数据表扫描（O(全部 manifest) 远程 IO）；MTMV refresh 还要重复 4-6 次 | 1/查询 × O(manifests)；4-6/refresh | 所有分区 iceberg 表 | C7 C23 C22 | +| **P0** | 簇2 #64134 复活 | `file_format_type` 兜底走**整表 planFiles()**，每查询 1-2 次，且与 planScan 自己的枚举完全冗余 | 1-2/查询 × 整表 manifest 扫描 | 无 write-format 属性的表（迁移表等） | C2 C11 | +| **P1** | C9 | information_schema.tables / SHOW TABLE STATUS 循环内**每表一次远程 loadTable**（只为拿 comment） | N 表 × loadTable 串行 | BI 工具高频路径 | C9 | +| **P1** | C3 | REST vended-credentials 下 **每个 data file / delete file** 重建 StorageProperties + hadoop Configuration | O(N_files+N_deletes) × Conf 构建 | REST + vended credentials | C3 | +| **P1** | 簇4 manifest cache 旁路 | streaming 路径与 COUNT(*) 下推路径都**绕过已开启的 IcebergManifestCache**（恰是大表/元数据查询最需要的场景） | 每查询整套 manifest 远程重读 | manifest cache 开启时 | C17 C18 | +| **P1** | C20 | 一条 DML 写语句 3-5 次远程 load 同一张表 | k≈3-5/语句 | 所有 iceberg 写 | C20 | +| **P2** | C5 | streaming pump 逐 split 入队：每 split 重建 backend 候选集 + 锁往返（fe-core 框架层） | per-split × 本地 CPU/锁 | ≥1024 文件的大扫描 | C5 | +| **P2** | C8 | 同一组 WHERE conjunct 被转换 5-6 次/查询 | k≈5-6 × 本地 CPU | 带谓词查询 | C8 | +| **P2** | 簇5 per-split 不变量 | buildRange 逐 byte-slice 重算分区 JSON/identity map/delete 转换；v3 delete 双重 thrift 转换；通用节点造完即弃的路径解析；不变 payload 逐 range 重复（RPC 膨胀） | per-split × 本地 CPU/字节 | 大 split 数 / MOR 表 | C12 C13 C14 C15 | +| **P2** | 维护路径 | rewrite_data_files **每 group 一次整表 planFiles**（G+1 次）；expire_snapshots 逐 snapshot×逐 manifest 读、零去重（S×M） | G/次、S×M/次 × 远程 IO | EXECUTE 维护命令 | C19 C21 | + +## 2. 各簇详述 + +### 簇1（P0）：无 Table 对象缓存 —— 一次规划 3~7 次远程 loadTable [C1 C4 C6 C10 C16] + +**根因**：`IcebergCatalogOps.loadTable`（`IcebergCatalogOps.java:340`）是对 SDK +`catalog.loadTable()` 的裸委派——每次调用都是一次 metastore RPC（HMS getTable / REST GET） ++ metadata.json 对象存储读取。全仓无 `CachingCatalog` 包装（grep 验证）； +`IcebergLatestSnapshotCache` 只缓存 `(snapshotId, schemaId)` 二元组，**不缓存 Table 对象**。 +于是 SPI 的每个入口各自 load： + +一次带 WHERE 的同步查询的 loadTable 计数（C6/C16 独立核对一致）： + +| # | 调用点 | 链路 | +|---|---|---| +| 1 | properties 计算①之 getColumnHandles | `FileQueryScanNode.init:139 → initSchemaParams:183 → PluginDrivenScanNode.getPathPartitionKeys:537 → getOrLoadPropertiesResult:1776 → buildColumnHandles:1780 → IcebergConnectorMetadata.getColumnHandles:587 → loadTable` | +| 2 | properties 计算①之 getScanNodeProperties | `getOrLoadPropertiesResult:1801 → IcebergScanPlanProvider.getScanNodeProperties:1300 → resolveTable:1981-1993 → loadTable` | +| 3-4 | properties 计算②（同上两次重来） | `convertPredicate:795-798` **无条件**清空 `cachedPropertiesResult` → `createScanRangeLocations:325 getFileFormatType` 触发整体重算（见 C1） | +| 5 | batch-mode 探测 | `FileQueryScanNode:376 isBatchMode → computeBatchMode:1414 → streamingSplitEstimate:410 resolveTable`（顺带再读一次 manifest-list，C4） | +| 6 | getSplits 再取列句柄 | `PluginDrivenScanNode.getSplits:1202 → buildColumnHandles → getColumnHandles:587` | +| 7 | planScan 正主 | `planScanInternal:562 resolveTable` | + +**放大细节（C1）**：`convertPredicate` 的失效是**一揽子**的——而 iceberg 连 `applyFilter` +override 都没有（谓词对 properties 的唯一影响是 pushdown-predicates 一个 prop）， +loadTable、format 解析、schema 字典 thrift+base64 编码、凭证 overlay 全部是 filter 不变量， +却整体重算一遍。 + +**影响**：每查询 +3~6 次冗余远程元数据往返（单次 50-300ms 量级）≈ 规划延迟 +0.2~1.5s， +点查/小查询被规划延迟支配；metastore/REST 服务 QPS 被放大 3~7 倍。 +legacy 对照：单一 source 缓存 Table，一次规划 1 次命中。 + +**修复方向**（verifier 建议汇总）：以 `beginQuerySnapshot` 已有的 pin 为天然 key， +做 **per-planning-pass 的 Table memo**（`(TableIdentifier, pinnedSnapshotId)` 键， +挂在长生命周期的 `IcebergConnector` 上或随 handle 传递），让 +`getColumnHandles / resolveTable / streamingSplitEstimate / planScan` 共享一次 load； +同时把 `convertPredicate` 的失效收窄到真正依赖 conjunct 的 prop。 + +### 簇2（P0）：#64134 原型在新框架复活 [C2 C11] + +**这是 JIRA 问题一的直系再现**。`getScanNodeProperties` 发 scan 级 `file_format_type` +（BE 靠它选 V1/V2 scanner，不能不发）： + +``` +PluginDrivenScanNode.getOrLoadPropertiesResult:1801 + → IcebergScanPlanProvider.getScanNodeProperties:1311 + → IcebergWriterHelper.getFileFormat(table):265 + → resolveFileFormatName:277（两个属性都未设时） + → inferFileFormatFromDataFiles:287 + → table.newScan().planFiles():291 ← 无过滤整表 manifest 扫描 +``` + +- **乘数**：每次 properties 计算 1 次 = 无谓词查询 1 次/查询、带谓词 2 次/查询 + （C1 的双计算叠加）。对比旧 bug 的 per-split，乘数降了，但**每条 SELECT/EXPLAIN 都要付**。 +- **纯冗余**：planScan 自己的 planFiles 枚举里 `dataFile.format()` 每个文件都有—— + 为了拿"第一个文件的格式"专门再扫一遍整表。 +- **门槛**：表属性无 `write-format` 且无 `write.format.default`。不只迁移表—— + 任何写入引擎从不显式设该属性的表都中（iceberg 的 parquet 默认值是读侧约定，不落属性）。 +- **修复方向**：按 `(table UUID, snapshotId)` memoize 解析结果（快照不变 ⇒ 格式不变）， + 或从 split 枚举结果反推 scan 级 format 传下去；配合簇1 的失效收窄消掉第二次。 + +### 簇3（P0）：分区视图每查询重建，无跨查询缓存 [C7 C23 C22] + +**根因**：分析期 `BindRelation.getLogicalPlan:733 → StatementContext.loadSnapshots:988-998 +→ PluginDrivenMvccExternalTable.loadSnapshot:343 → materializeLatest:126` 对分区表走 +`getMvccPartitionView / listPartitions → IcebergPartitionUtils.loadRawPartitions:709-718` += PARTITIONS 元数据表 `planFiles()` + `rows()`——SDK 聚合时要读该快照**全部** data+delete +manifest。`StatementContext` 只在**单语句内** memoize；跨查询无任何缓存 +（legacy 的二级分区缓存在 CACHE-P1 决策中被有意放弃——该决策的代价在这里显形）。 + +- **影响**：数千 manifest 的分区表：每条查询（哪怕 WHERE 全命中裁剪）在拿到 scan node + 之前先付 MB~GB 级 manifest 读 + 秒级延迟；随表历史规模而非查询选择性增长。 + 与簇1 叠加后很可能是分区大表规划延迟的第一大头。 +- **MTMV 放大（C22）**：一次 refresh 流程里 `isValidRelatedTable / alignMvPartition / + generateRelatedPartitionDescs / getAndCopyPartitionItems...` 各自 materialize 同一视图 + 4-6 次（无 refresh 级 pin），还引入枚举点之间的快照偏移风险。 +- **修复方向**：按 `(TableIdentifier, snapshotId)` 缓存分区视图（pin 在 + `beginQuerySnapshot` 后已在 handle 上，key 天然可用），挂 fe-connector-cache； + MTMV 侧在 `MTMVRefreshContext` 加 refresh 级 MvccSnapshot pin。 + +### C9（P1）：information_schema.tables 循环内每表 loadTable + +`FrontendServiceImpl.listTableStatus:719 for (TableIf table : tables)` → +`:755 setComment(table.getComment())`（**无条件**，不看请求是否需要 comment 列）→ +`PluginDrivenExternalTable.getComment:944 → IcebergConnectorMetadata.getTableComment:305 → +loadTable`。N 张表 = N 次串行远程 load：几百张表的库一条 +`SELECT * FROM information_schema.tables` 要几十秒到分钟级，且 BI 工具高频触发。 +教科书级"伪装成轻访问器"（getComment ← 远程 IO）。 +**修复**：建表/schema-cache 时捕获 comment 随表对象缓存，或该路径按需惰性取。 + +### C3（P1）：vended-credentials 每文件重建 StorageProperties + +`buildRange:1105 normalizeUri` / `convertDelete:1157` → `DefaultConnectorContext.normalizeStorageUri:392-409 → buildVendedStorageMap:225-242 → StorageProperties.createAll`（遍历所有 provider + 构建 hadoop `Configuration` + 逐 key set）——**每个 data file 和每个 delete file 各一次**，而 vended token 在整个 scan 内不变。50k 文件 ≈ 数十秒纯 FE CPU。门槛：REST + `iceberg.rest.vended-credentials-enabled=true`（MOR 表加倍）。 +**修复**：token→typed-map 的推导按 scan 提升/在 `DefaultConnectorContext` 内做单条目 memo（token 恒等键）。 + +### 簇4（P1）：IcebergManifestCache 旁路 [C17 C18] + +manifest cache（`meta.cache.iceberg.manifest.enable`，默认 off）只接在同步路径 +`planFileScanTask:1681`（gate `isManifestCacheEnabled:1832`）上： + +- **C17**：文件数 ≥ `num_files_in_batch_mode`（默认 1024）时走 streaming + `streamSplits:449 → scan.planFiles():463`——**恰好把 cache 瞄准的大表全体踢出缓存** + （legacy 在 batch 模式下仍走 cache）。 +- **C18**：COUNT(*) 下推 `planScanInternal:594-598 → planCountPushdown:1021 → + scan.planFiles():1024` 在到达 cache 分支之前 return——元数据即可回答的查询反而全量远程读 + manifest（且 `ParallelIterable` 会激进提交所有 manifest 读任务，虽然只要第一个 task）。 + +**修复**：cache 开启时 streaming 判定返回 -1 退回同步物化路径（一行 legacy-parity 修复）； +count 分支改走 `planFileScanTask`。 + +### C20（P1）：写路径 3-5 次 load 同一张表 + +`PhysicalIcebergMergeSink.getRequirePhysicalProperties:161→198`、 +`PhysicalPlanTranslator.visitPhysicalConnectorTableSink:675,703`、 +`PluginDrivenTableSink.bindDataSink:175` → `IcebergWritePlanProvider.resolveTable:689-702` / +`beginWrite`（内含 tableExists ×2 + 无条件 refresh）。每条 DML +3~5 次串行远程往返。 +**修复**：语句级 resolve 一次传递；exists 从 load 结果推导。 + +### P2 组（CPU/payload/维护路径，影响门槛高但模式典型） + +- **C5**（fe-core 框架层，惠及所有连接器）：`startStreamingSplit:1638-1642` 逐 split + `addToQueue` → `SplitAssignment` 每 split 重建 backend 候选集 + （`FederationBackendPolicy.computeScanRangeAssignment:225-235` 全量 backend 拷贝 + + shuffle + multimap）+ `synchronized` 往返。10⁵-10⁶ split × ~100 BE ≈ 10⁷-10⁸ 冗余操作。 + 修复：pump 侧微批（64-256 个/批）。 +- **C8**：同一组 conjunct 在 `buildRemainingFilter`（×3 处）+ `buildScan:974` + + `getScanNodeProperties:1428`（EXPLAIN 专用序列化，**非 EXPLAIN 也跑**）被转换 5-6 次。 + 单次微秒~毫秒级，但与簇1 同源叠加。修复：node 字段 memo，与 `cachedPropertiesResult` + 同点失效。 +- **C12**：一个 data file 被 `TableScanUtil.splitFiles` 切成 k 个 byte-slice 后， + `buildRange:1045` 对每个 slice 重算 partition JSON（Jackson 序列化 + 时区格式化）、 + identity map、delete 转换——(specId, PartitionData) 级不变量。100k split ≈ 0.5-2s CPU。 +- **C13**：v3 rewritable-delete stash：同一 delete 列表 plan 期 `rewritableDeleteDescs:302-313` + 转一次 thrift，`populateRangeParams` 再转一次；每 slice 重复 put 相同 supply。 +- **C14**：通用节点 per-split 重复 `LocationPath.of`（URLEncoder+URI.create）、 + 重建 columns-from-path 却被 `IcebergScanRange.populateRangeParams:435-437` unset 丢弃； + `resolveScanProvider` 每 split 经 `getFileCompressType` 反复解析。 +- **C15**（payload 版放大）：同一 data file 的完整 delete 列表 + partition JSON 复制进 + **每个** byte-slice 的 `TFileRangeDesc`；共享 delete file 逐 data file 重复。大 MOR 扫描 + 计划体积多出 MB 级（FE 构建 + BE 解析双向付费）。 +- **C19**：`ConnectorRewriteDriver.run STEP3:143` 每个 rewrite group 调一次 + `registerRewriteSourceFiles` → 每次一遍 `useSnapshot(...).planFiles():379-383`。 + G 个 group = G+1 次整表扫描，G~50-200 时分钟级。修复：union 所有 group 一次注册 + （SPI 本来就收 `Set`）。 +- **C21**：`IcebergExpireSnapshotsAction.buildDeleteFileContentMap:271-293` 对**每个** snapshot + 读其全部 delete manifest，无 visited-path 去重——相邻 snapshot 的 manifest 大量重叠， + S×M 串行远程读。按 `ManifestFile.path()` 去重即坍缩为 O(distinct)。 + +## 3. 驳回项与旁获发现 + +- **驳回 R1**："字典新鲜度 poll 每 5s 重建分区视图"——重操作属实，但乘数驱动不成立： + `CreateDictionaryInfo.validateAndSet:164` 把源表强转 `org.apache.doris.catalog.Table`， + 而 ExternalTable 只实现 `TableIf` ⇒ **对外表 CREATE DICTIONARY 在校验期就 + ClassCastException**，poll 路径不可达。 + ⚠️ 旁获：这本身可能是一个功能缺口/待修的强转 bug（外表字典完全不可用），与本审计无关， + 单独记录。 +- 去重合并说明：33→24 的合并里,同根因跨 lens 的发现（如 loadTable 簇）被保留为独立条目 + 以互为佐证，报告中按簇归并陈述。 + +## 4. 结构性观察（为什么会长出这一类问题） + +1. **SPI 入口各自为政取 Table**：`ConnectorMetadata` / `ScanPlanProvider` / 写侧 provider + 的每个方法都独立 `resolveTable/loadTable`，接口上没有"本次规划已解析的 Table"这一共享 + 载体；legacy 靠 source 对象天然持有。→ 修复应该是框架级的（per-planning-pass 载体或 + connector 级 snapshot-keyed cache），否则每加一个 SPI 方法就多一次 load。 +2. **成本契约缺失复刻了 #64134 的成因**：`getFileFormat / getComment / getScanNodeProperties` + 这类名字读不出"远程 IO"；调用方（含 fe-core 通用节点）按 O(1) 使用。审计确认的 23 条里 + 过半是这个模式。建议在 SPI javadoc 里显式标注成本等级（cheap/loaded-metadata/remote-IO）， + 新代码 review 按此把关。 +3. **失效粒度过粗**：`convertPredicate` 一揽子清缓存是"正确但昂贵"的懒办法，直接把簇1、 + 簇2 的成本翻倍。 +4. **缓存修在旧路径上**：manifest cache 只挂在同步 materialize 路径，streaming/count 两条 + 新路径没接——加新路径时没有"必须过一遍既有缓存清单"的动作。 + +## 5. 建议的修复优先级（供 review 讨论，非结论） + +1. per-planning-pass Table memo（簇1，一并收窄 convertPredicate 失效）——单点改动收益最大； +2. 分区视图 `(table, snapshotId)` 缓存 + MTMV refresh pin（簇3）； +3. `file_format_type` 兜底 memoize / 从枚举反推（簇2）； +4. manifest cache 两条旁路接回（簇4，其中 C17 有一行式 legacy-parity 修法）； +5. C9 / C3 / C20 各自局部 hoist； +6. P2 组择机随重构顺手处理（C19/C21 改动极小收益明确，可先行）。 + +—— 以上待 review。任何一条立项前建议先按 findings.json 里的调用链复核一遍行号。 From b1551b29973762a170cf022ba3369f21bca60f70 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 17 Jul 2026 16:53:02 +0800 Subject: [PATCH 02/47] [doc](catalog) add fe-connector-iceberg hot-path perf fix task workspace MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Set up an independent task workspace (plan-doc/perf-hotpath-iceberg/) to drive the fixes for the 2026-07-17 fe-connector-iceberg hot-path heavy-op audit. It groups the 23 confirmed findings (C1-C23) into 11 tasks at cluster granularity (one cluster = one task), ordered by the audit's §5 fix priority. Contents: - README.md: orientation, per-task workflow (aligned with step-by-step-fix), and the iron-rule constraints (fe-core source-only-shrinks, connector-side memo, no source-specific code in generic nodes). - tasklist.md: 11 PERF-NN tasks with an at-a-glance overview table and full per-finding traceability. - HANDOFF.md: rolling per-session handoff, entry point set to PERF-01. - progress.md: append-only log. - designs/: placeholder for per-task design/summary docs (created per fix). Analysis is not duplicated here; it points to the existing audit report, findings.json, and problem-class doc. No product code changed. Co-Authored-By: Claude Opus 4.8 (1M context) --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 50 +++++++++ plan-doc/perf-hotpath-iceberg/README.md | 82 ++++++++++++++ .../perf-hotpath-iceberg/designs/README.md | 11 ++ plan-doc/perf-hotpath-iceberg/progress.md | 14 +++ plan-doc/perf-hotpath-iceberg/tasklist.md | 102 ++++++++++++++++++ 5 files changed, 259 insertions(+) create mode 100644 plan-doc/perf-hotpath-iceberg/HANDOFF.md create mode 100644 plan-doc/perf-hotpath-iceberg/README.md create mode 100644 plan-doc/perf-hotpath-iceberg/designs/README.md create mode 100644 plan-doc/perf-hotpath-iceberg/progress.md create mode 100644 plan-doc/perf-hotpath-iceberg/tasklist.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md new file mode 100644 index 00000000000000..004f3020f4de49 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -0,0 +1,50 @@ +# 🤝 Session Handoff — fe-connector-iceberg 热路径重操作修复 + +> **滚动文档**：每次 session 结束**覆盖式更新**，只保留下一个 session 必须的上下文；完成明细进 `git log` + `designs/` + `progress.md`，别堆这里。 +> **范围** = 逐一修复审计报告 `../reviews/perf-audit-fe-connector-iceberg-2026-07-17.md` 的 23 条发现（C1–C23 → **一簇一任务** 11 个 `PERF-NN` 任务，见 [`tasklist.md`](./tasklist.md)）。 +> 开场必读顺序、单项立项流程、约束铁律：[`README.md`](./README.md)。 + +--- + +# 🆕 下一个 session = **启动 PERF-01（簇1，收益最大）** + +## 现状（session 0，2026-07-17） + +- 本任务空间**刚建好**：`README.md` / `tasklist.md` / `HANDOFF.md` / `progress.md` / `designs/` 就位。 +- 粒度已定：**一簇一任务**（用户 2026-07-17 拍板）—— 23 发现归并为 **11 个任务**。 +- **零任务已实现** —— 全部 11 项 `⏳ 待启动`。 +- 审计报告是**待 review 的结论草案**（报告 §5 自称"供 review 讨论，非结论"）。因此**不照单硬修**：每项第一步是**按 findings.json 的调用链复核行号 + 乘数**（README §单项立项流程 step 2），复核站得住再设计实现。 + +## 下一个 session 第一件事（精确到动作） + +1. 向用户复述并确认起点："上次建好了 iceberg 热路径修复任务空间，下一步启动 **PERF-01（簇1，per-planning-pass Table memo）**，对吗？"（或用户另指定某项，如想先做改动小的 **PERF-08**＝维护路径 C19/C21）。 +2. 确认后，**先复核 PERF-01**（design subagent）： + - `grep` 复核簇1 的 7 个 loadTable 调用点行号（报告簇1 表 / findings.json 里 C1 C4 C6 C10 C16 的 `heavy_op` + `multiplicity` 字段）：`IcebergCatalogOps.java` 的 `loadTable`（报告记 :340）、`IcebergConnectorMetadata.getColumnHandles`（:587）、`IcebergScanPlanProvider.getScanNodeProperties`（:1300/1311）+ `resolveTable`（:1981-1993）、`convertPredicate`（:795-798 无条件清 `cachedPropertiesResult`）、`streamingSplitEstimate`（:410）、`planScanInternal`（:562）。 + - 确认无 `CachingCatalog`（全仓 grep）、`IcebergLatestSnapshotCache` 确只存 `(snapshotId,schemaId)`。 + - 确认 `beginQuerySnapshot` 的 pin 可作 `(TableIdentifier, snapshotId)` memo key。 +3. 复核站得住 → 写 `designs/FIX-PERF-01-table-memo-design.md`（含**度量方案**：用调用计数证明 loadTable 从 N→1），设计红队，再实现。 + +## ⚠️ 关键认知（别重踩） + +- **PERF-01 必须最先做**：它的 `convertPredicate` 失效收窄是 PERF-03（簇2 消第二次计算）和 PERF-10（C8）的前置；它立住的 `(table, snapshotId)` memo 模式被 PERF-02/03 复用。顺序错了要返工。 +- **这是性能修复 = 行为必须不变**：验收闸门是「相关 UT/e2e 全绿（parity）+ 证明减负（调用计数/时延）」，不是新增功能。每项尽量补一个**调用计数守门**（如断言规划期 loadTable=1），否则回归无法自动发现。 +- **memo 放连接器侧，别碰 fe-core 源**（fe-core 只出不进铁律）。PERF-09（C5 微批）、PERF-11 内 C14（通用节点）虽改 fe-core 通用层（允许，因非 source-specific），但须证跨连接器 byte+cost 双不变、禁按源名分支。 +- **审计草案可能被复核推翻**：若某条复核后乘数/路径不成立，tasklist 标 🔬 + 记 progress，**不硬凑修复**。 + +--- + +# 🧰 构建/验证坑（承自主线 HANDOFF，本空间直接复用，别再踩） + +1. **maven build cache 会静默跳过 surefire** —— 日志 `Skipping plugin execution (cached): surefire:test` 时 **BUILD SUCCESS 是空的**（报告是陈旧文件）。**所有测试必须加 `-Dmaven.build.cache.enabled=false`**。 +2. **`mvn ... | tail` 后的 `$?` 是 `tail` 的**，不是 maven 的 —— 重定向到文件再取 `$?`，或读 `BUILD SUCCESS`/`BUILD FAILURE` 行。 +3. **maven 用绝对 `-f /fe/pom.xml`**（cwd 跨调用持久，`cd` 会破相对路径 / 触发权限弹窗）。 +4. **`regression-test/conf/regression-conf.groovy` 工作区本就是脏的**（session 开始前即 `M`）—— **别顺手 `git add -A`**，按需精确 add。 +5. **并发 session 共享同一 worktree** —— 动码前探测并发活动（`git log/status` + maven 进程 + 近 90s mtime）；`pgrep maven` 可能查到 1 天前的僵尸 until-loop，**看 `etime` 再判**，别误判成活跃 session 无谓停手。小步快提交缩短被 amend 卷走窗口。 +6. **本 worktree 的 `.git` 是文件不是目录**（linked worktree）—— rebase/冲突脚本禁硬编码 `.git/rebase-merge`，一律 `git rev-parse --git-path`。 + +--- + +# 🗂 开放问题 / 待用户裁决（不是 TODO，是需要拍板的） + +- **审计整体是否已被用户接受为立项依据**：目前按"每项立项前自行复核行号/乘数"处理。若用户已整体签字，可省去部分复核力度（但行号复核仍建议保留）。 +- ~~任务粒度~~：已定 **一簇一任务**（2026-07-17），无遗留。 diff --git a/plan-doc/perf-hotpath-iceberg/README.md b/plan-doc/perf-hotpath-iceberg/README.md new file mode 100644 index 00000000000000..72d9d6fe21e57f --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/README.md @@ -0,0 +1,82 @@ +# 📦 任务空间 — fe-connector-iceberg 热路径重操作修复 + +> **独立任务空间**，与 catalog-spi 主线（`plan-doc/HANDOFF.md`，当前占用者 = CI-997422 任务线）**并行但不混流**。 +> 目标：把 [`../reviews/perf-audit-fe-connector-iceberg-2026-07-17.md`](../reviews/perf-audit-fe-connector-iceberg-2026-07-17.md) 审计出的 +> **23 条确认发现（C1–C23）逐一修复**，消掉"新框架 SPI 各入口各自 loadTable/各自扫" 造成的重操作放大。 +> 协作规范沿用 [`../AGENT-PLAYBOOK.md`](../AGENT-PLAYBOOK.md)。 + +--- + +## 🚩 一句话背景（2026-07-17 审计结论） + +**总病灶**：新框架的 SPI 各入口"每次自己 loadTable / 自己扫"，而 legacy 有单一 source 持缓存 Table —— +同一信息在一次规划里被远程重取 **3~7 次**；另有 #64134 的 planFiles 兜底在新框架以 per-query 乘数复活。 +审计分 **P0/P1/P2 三层、七簇**，55-agent 对抗审计 + 人工抽验，**23 确认 / 1 驳回 / 0 存疑**。 + +> ⚠️ **审计报告是"待 review"的结论草案**（报告 §5「供 review 讨论，非结论」）。故每项**立项第一步是复核**（见下 §立项流程 step 2），不是照单实现。 + +--- + +## 📂 本空间文件 + +| 文件 | 用途 | 更新方式 | +|---|---|---| +| [`tasklist.md`](./tasklist.md) | **Task list** —— 唯一进度总览，`PERF-NN` 勾选 + 状态表 | 每完成一项随 commit 勾 `[x]` + 填状态/commit | +| [`HANDOFF.md`](./HANDOFF.md) | **交接文档** —— 只写「下一个 session 第一件事做什么」 | 每 session 结束**覆盖式**更新 | +| [`progress.md`](./progress.md) | **进度记录** —— append-only 日志（日期 / commit / 结论 / 踩坑） | 只追加，不覆盖 | +| [`designs/`](./designs/) | **per-task 设计 + 小结** —— `FIX-PERF-NN--design.md` / `-summary.md` | 每项**修复时**逐个建，不预建 | + +> **分析/设计的权威来源不在本空间**（避免重复记录），而是： +> 审计报告 `../reviews/perf-audit-...md`（分层结论 + 各簇详述 + §5 修复优先级） +> · 证据 JSON `../reviews/perf-audit-...-findings.json`（**每条**发现的完整调用链 + 双路对抗验证意见，是复核行号的第一手材料） +> · 问题类 `../perf-heavy-op-hot-path-problem-class.md`（三要素 + A/B/C/D 变体 + 审计清单）。 + +--- + +## ▶️ 新 session 开场流程（必须遵守） + +``` +1. Read plan-doc/perf-hotpath-iceberg/HANDOFF.md ← 上次留言 + 下一步 +2. Read plan-doc/perf-hotpath-iceberg/tasklist.md ← 勾到哪了、下一个 PERF-NN 是谁 +3. 需要某条发现的细节时才 Read 审计报告对应簇 / findings.json 对应条目（别默认全读） +4. 用一句话向用户复述："上次做完了 X，下一步是 PERF-NN，对吗？" +5. 等用户确认后按下面「单项立项流程」开始 +``` + +**⚠️ 行号信 HEAD 不信文档** —— 本空间与审计报告所有 `file:line` 是 **2026-07-17 / 分支 `catalog-spi-review-16`** 基线，代码动了就以 `grep` 为准。 + +--- + +## 🔁 单项立项流程（每个 PERF-NN，对齐 `step-by-step-fix` skill） + +> 一次一个任务，串行推进。这些是**性能修复 → 必须行为不变**，所以复核与"证明只减不改"是重点。 + +1. **确认 owner 单一**：同一时刻一个任务只一个 session 在做（并发探测见 HANDOFF「构建/并发坑」）。 +2. **复核发现（design subagent 先做，动码前）**：按 findings.json 的调用链**重新 grep 行号**、重新确认①重操作真实成本 ②乘数（per-query / per-split / per-file）③是否真无缓存兜住。审计是草案，复核**可能推翻或缩小**某条 —— 如推翻，在 tasklist 标 🔬 并记 progress，不硬修。 +3. **写设计文档** `designs/FIX-PERF-NN--design.md`：Problem / Root Cause / Design / Implementation Plan / Risk / Test Plan（含如何**度量收益**：loadTable/planFiles 调用计数、EXPLAIN 时延、或 debug 计数器）。 +4. **设计红队**（对抗 review，符合本项目 clean-room 偏好）：至少一个独立视角挑刺后再实现。 +5. **实现**：最小改动、守约束（见下铁律）、保持风格。 +6. **验证**（这是性能修复，闸门 = 行为 parity + 证明减负）： + - 相关 **UT / e2e 全绿**（parity，禁回归）；必要时补一个**调用计数断言**当守门（如"规划期 loadTable 从 N 降到 1"）。 + - build 用 `-Dmaven.build.cache.enabled=false`（否则 surefire 被静默跳过，见 HANDOFF）。 +7. **独立 commit**：`[perf](catalog) fe-connector-iceberg: (PERF-NN)`，正文写 root cause / 修法 / 度量数 / 守门。 +8. **写小结** `designs/FIX-PERF-NN--summary.md`（Problem/Root Cause/Fix/Tests/Result）。 +9. **更新 `tasklist.md`**（勾 `[x]` + 填 commit/状态）→ **追加 `progress.md`** → **覆盖 `HANDOFF.md`**（下一项是谁）。 + +--- + +## 🧱 约束铁律（**违反即返工，动码前记牢**） + +1. **fe-core 源只出不进 + 禁 scaffolding 搬迁**（本项目删旧代码期两条铁律）：修复的缓存/memo 默认放**连接器侧**（`IcebergConnector` / handle / `fe-connector-cache`），**不得**为图省事把 Table 缓存、属性解析、派生 helper 塞进 fe-core。PERF-01/02/03 的 memo 都挂连接器。 +2. **fe-core 通用节点保持 connector-agnostic**：PERF-09（fe-core 框架层微批）、PERF-11 内的 C14（通用节点 per-split）改的是**通用**逻辑 —— 允许改（惠及所有连接器、非 source-specific），但**禁**按 iceberg/源名分支；且属共享热路径，须证 **byte + cost 对所有连接器双不变**。 +3. **fe-core 不解析属性**：任何属性相关派生放插件侧组装 → BE thrift → 回传，别在 fe-core 加解析。 +4. **失效收窄要精确**：PERF-01 收窄 `convertPredicate` 失效时，只保留真正依赖 conjunct 的 prop（= pushdown-predicates 一个）；错杀会把簇1/簇2 成本翻回去。 +5. **快照 pin 是缓存 key 的天然来源**：PERF-01/02/03 统一用 `beginQuerySnapshot` 已 pin 的 `(TableIdentifier, snapshotId)` 做 key —— 先做 PERF-01 立住这套 key/memo 模式，02/03/10 复用，别各造一套。 + +--- + +## 🔗 与其它文档/任务的关系 + +- **主线** `plan-doc/HANDOFF.md`：当前是 CI-997422 任务线，本任务在其「📎 并行独立任务：热路径重操作审计」小节里被引出 —— 本空间即那条线的落地执行区。**两者不混流**。 +- **问题类可复用**：审计结论里「其余连接器（hive/paimon/hudi/mc）按同一问题类 + 同一 workflow 逐个审计」是后续工作 —— 届时各连接器建**各自的兄弟任务空间**（对齐本项目 per-connector workspace 惯例），不并进本空间。 +- 协作规范、context 预算、subagent/handoff 纪律：一律沿用 [`../AGENT-PLAYBOOK.md`](../AGENT-PLAYBOOK.md)。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/README.md b/plan-doc/perf-hotpath-iceberg/designs/README.md new file mode 100644 index 00000000000000..be9aa9de646242 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/README.md @@ -0,0 +1,11 @@ +# per-task 设计 + 小结 + +每个 `PERF-NN` 任务在**实现时**（不预建）产出两份文档： + +- `FIX-PERF-NN--design.md` —— Problem / Root Cause / Design / Implementation Plan / Risk / Test Plan（含收益度量方案） +- `FIX-PERF-NN--summary.md` —— Problem / Root Cause / Fix / Tests / Result + +命名对齐仓库既有惯例（`plan-doc/designs/FIX-*-design.md`、`plan-doc/tasks/designs/FIX-*-design.md`）。 +`` 用短横线短语，例：`FIX-PERF-01-table-memo-design.md`。 + +立项流程见 [`../README.md`](../README.md) §单项立项流程。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md new file mode 100644 index 00000000000000..feb2b50ba1709e --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -0,0 +1,14 @@ +# Progress Log — fe-connector-iceberg 热路径重操作修复 + +> **Append-only** 日志：只追加、不覆盖（覆盖式状态在 HANDOFF/tasklist）。 +> 每完成 / 阻塞 / 复核推翻 / 重大发现加一段：日期 · 任务 · commit · 结论 · 踩坑。 + +--- + +## 2026-07-17 — session 0：建任务空间 + +- 依据审计报告 `../reviews/perf-audit-fe-connector-iceberg-2026-07-17.md`（23 确认发现，P0/P1/P2 三层七簇）建立独立任务空间 `plan-doc/perf-hotpath-iceberg/`。 +- 落地文件：`README.md`（导航 + 单项立项流程 + 约束铁律）、`tasklist.md`（23 发现 → PERF-NN 任务，按 §5 优先级排序 + 总览表）、`HANDOFF.md`（起点 = PERF-01）、`progress.md`（本文件）、`designs/`（per-task 设计/小结空目录）。 +- 任务粒度：**一簇一任务**（用户 2026-07-17 拍板）—— 按审计 §1 总览表行边界，簇级并成一项、未成簇的独立发现各一项，共 **11 个任务**。全部 `⏳ 待启动`，零实现。 +- 未动任何产品代码；审计报告仍为待 review 草案，约定每项立项前复核行号/乘数。 +- **下一步**：见 HANDOFF —— 启动 PERF-01（簇1，per-planning-pass Table memo）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md new file mode 100644 index 00000000000000..51d870daf27788 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -0,0 +1,102 @@ +# Task List — fe-connector-iceberg 热路径重操作修复 + +> **唯一进度清单**。每完成一项，随 commit 把对应行改 `[x]` 并在总览表填状态/commit。 +> ID 一旦分配**永不复用**；删除的任务标 `[deleted YYYY-MM-DD <原因>]` 保留占位。 +> 权威分析（别在本文件复制）：审计报告 [`../reviews/perf-audit-fe-connector-iceberg-2026-07-17.md`](../reviews/perf-audit-fe-connector-iceberg-2026-07-17.md) +> · 证据 JSON [`../reviews/perf-audit-fe-connector-iceberg-2026-07-17-findings.json`](../reviews/perf-audit-fe-connector-iceberg-2026-07-17-findings.json) +> · 问题类 [`../perf-heavy-op-hot-path-problem-class.md`](../perf-heavy-op-hot-path-problem-class.md)。 +> 每项立项流程、约束铁律、验收口径见 [`README.md`](./README.md) —— **动码前必读**。 + +## 总览（at-a-glance） + +- 覆盖：审计 **23 条确认发现（C1–C23）** → **一簇一任务**归并为 **11 个可提交修复任务**（按审计 §1 总览表行边界：簇级并成一项，未成簇的独立发现各一项）。1 条驳回（R1，见报告 §3）不立项。 +- **推荐顺序 = 审计 §5 优先级**（下表自上而下）。P0 收益最大先做；PERF-08（维护路径 C19/C21）改动小、收益明确，可作热身先行插队。 +- 状态图例：⏳ 待启动 · 🚧 进行中 · ✅ 完成 · ❌ 阻塞（备注写原因） · 🔬 复核中（re-verify 发现行号/乘数已变，待重判） + +| ID | 优先级 | 覆盖发现 | 主题（一句话） | 依赖 | 状态 | commit | +|---|---|---|---|---|---|---| +| PERF-01 | P0 | C1 C4 C6 C10 C16 | 一次规划 3~7 次远程 loadTable → per-planning-pass Table memo + 收窄 convertPredicate 失效 | — | ⏳ | | +| PERF-02 | P0 | C7 C22 C23 | 分区视图每查询重扫 PARTITIONS 元数据表 → `(table,snapshotId)` 缓存 + MTMV refresh pin | 与 01 共享快照 pin 机制 | ⏳ | | +| PERF-03 | P0 | C2 C11 | #64134 复活：`file_format_type` 兜底走整表 planFiles → memoize / 从枚举反推 | 受益于 01 的失效收窄（消第二次） | ⏳ | | +| PERF-04 | P1 | C17 C18 | streaming / COUNT(*) 下推旁路 IcebergManifestCache → 两条旁路接回 | — | ⏳ | | +| PERF-05 | P1 | C9 | information_schema.tables 循环内每表 loadTable（只为拿 comment） → 随表缓存 / 惰性取 | — | ⏳ | | +| PERF-06 | P1 | C3 | REST vended-cred 每 data/delete file 重建 StorageProperties+Configuration → scan 级 memo | — | ⏳ | | +| PERF-07 | P1 | C20 | 一条 DML 3~5 次 load 同表 → 语句级 resolve 一次传递 | — | ⏳ | | +| PERF-08 | P2 | C19 C21 | 维护路径逐单位重扫 / 无去重（rewrite_data_files 每 group planFiles；expire_snapshots S×M）→ union 一次注册 / 按 path 去重 | 改动小可先行 | ⏳ | | +| PERF-09 | P2 | C5 | **（fe-core 框架层）** streaming pump 逐 split 重建 backend 候选集 + 锁往返 → 微批 | ⚠ 跨连接器，见约束 | ⏳ | | +| PERF-10 | P2 | C8 | 同组 WHERE conjunct 被转换 5~6 次/查询 → node 字段 memo（与 01 同点失效） | 与 01 同源 | ⏳ | | +| PERF-11 | P2 | C12 C13 C14 C15 | per-split 不变量重算 + payload 逐 slice 复制 → hoist / (specId,PartitionData) memo / per-file 共享 | 同区域批量 · ⚠ 含 fe-core 通用节点 | ⏳ | | + +--- + +## P0 — 无缓存/兜底类，触发面=所有 iceberg 查询 + +### [ ] PERF-01 — 簇1：per-planning-pass Table memo（C1 C4 C6 C10 C16） +- **病灶**：`IcebergCatalogOps.loadTable:340` 是对 SDK `catalog.loadTable()` 的裸委派（每次 = metastore RPC + metadata.json 读）；全仓无 `CachingCatalog`，`IcebergLatestSnapshotCache` 只存 `(snapshotId,schemaId)` 不存 Table。SPI 各入口各自 load ⇒ 一次带 WHERE 的查询 **3~7 次** loadTable（7 个调用点见报告簇1 表）。放大器 = `convertPredicate:795-798` **无条件**清空 `cachedPropertiesResult` ⇒ 整份 properties（loadTable/format/schema thrift+base64/凭证 overlay，全是 filter 不变量）第二次重算。 +- **修复方向**：以 `beginQuerySnapshot` 已有的 pin 为天然 key，做 per-planning-pass 的 **Table memo**（键 `(TableIdentifier, pinnedSnapshotId)`，挂长生命周期 `IcebergConnector` 或随 handle 传递），让 `getColumnHandles / resolveTable / streamingSplitEstimate / planScan` 共享一次 load；同时把 `convertPredicate` 失效**收窄**到真正依赖 conjunct 的 prop（谓词对 properties 的唯一影响仅 pushdown-predicates 一个 prop）。 +- **收益**：每查询 −3~6 次远程往返 ≈ 规划延迟 −0.2~1.5s；metastore/REST QPS 除以 3~7。 +- **约束**：memo 放**连接器侧**，**不得**在 fe-core 加 Table 缓存或属性派生（见 README 铁律）。 +- **依赖**：无。**建议第一个做** —— 收益最大，且其失效收窄是 PERF-03、PERF-10 的前置。 + +### [ ] PERF-02 — 簇3：分区视图跨查询缓存 + MTMV refresh pin（C7 C22 C23） +- **病灶**：分析期 `loadSnapshot → materializeLatest:126` 对分区表走 `IcebergPartitionUtils.loadRawPartitions:709-718` = PARTITIONS 元数据表 `planFiles()+rows()`（SDK 聚合读该快照**全部** data+delete manifest）。`StatementContext` 只在单语句内 memoize，跨查询零缓存（legacy 二级分区缓存在 CACHE-P1 决策中被有意放弃）。MTMV 放大（C22）：一次 refresh 里 `isValidRelatedTable/alignMvPartition/generateRelatedPartitionDescs/getAndCopyPartitionItems` 各 materialize 同视图 **4~6 次**（无 refresh 级 pin，且引入枚举点间快照偏移风险）。 +- **修复方向**：按 `(TableIdentifier, snapshotId)` 缓存分区视图（pin 在 `beginQuerySnapshot` 后已在 handle 上），挂 `fe-connector-cache`；MTMV 侧在 `MTMVRefreshContext` 加 refresh 级 `MvccSnapshot` pin。 +- **依赖**：与 PERF-01 共享 `(table, snapshotId)` 快照 pin —— 先做 01 立住模式，02 复用。 + +### [ ] PERF-03 — 簇2：#64134 planFiles 兜底复活（C2 C11） +- **病灶**：`getScanNodeProperties:1311 → IcebergWriterHelper.getFileFormat:265 → resolveFileFormatName:277 → inferFileFormatFromDataFiles:287 → table.newScan().planFiles():291`（无过滤整表 manifest 扫描）。门槛=表属性无 `write-format` 且无 `write.format.default`（含迁移表 + 任何写引擎从不显式设该属性的表）。纯冗余：planScan 自己的 planFiles 枚举里每个 `dataFile.format()` 都有，为拿"第一个文件格式"专门再扫全表。乘数=每次 properties 计算 1 次（无谓词 1/查询、带谓词 2/查询，叠 C1）。 +- **修复方向**：按 `(table UUID, snapshotId)` memoize 解析结果（快照不变 ⇒ 格式不变），或从 split 枚举结果反推 scan 级 format 传下去；配合 PERF-01 的失效收窄消掉第二次。 +- **依赖**：PERF-01 的 convertPredicate 收窄消除第二次计算。 + +--- + +## P1 — 局部旁路/局部 hoist，触发面较窄或特定场景 + +### [ ] PERF-04 — 簇4：IcebergManifestCache 两条旁路接回（C17 C18） +- **病灶**：manifest cache（`meta.cache.iceberg.manifest.enable`，默认 off）只接在同步路径 `planFileScanTask:1681`（gate `isManifestCacheEnabled:1832`）。**C17**：文件数 ≥ `num_files_in_batch_mode`（默认 1024）走 streaming `streamSplits:449 → scan.planFiles():463`，**恰好把 cache 瞄准的大表踢出缓存**（legacy batch 模式仍走 cache）。**C18**：COUNT(*) 下推 `planScanInternal:594-598 → planCountPushdown:1021 → scan.planFiles():1024` 在到 cache 分支前 return（`ParallelIterable` 还激进提交所有 manifest 读任务，虽只要第一个）。 +- **修复方向**：cache 开启时 streaming 判定返回 -1 退回同步物化路径（**一行 legacy-parity 修复** = C17）；count 分支改走 `planFileScanTask`（C18）。 + +### [ ] PERF-05 — C9：information_schema.tables 每表 loadTable +- **病灶**：`FrontendServiceImpl.listTableStatus:719 for(table)` → `:755 setComment(table.getComment())`（**无条件**，不看请求是否要 comment 列）→ `PluginDrivenExternalTable.getComment:944 → IcebergConnectorMetadata.getTableComment:305 → loadTable`。N 表 = N 次串行远程 load；几百表的库一条 `SELECT * FROM information_schema.tables` 数十秒~分钟，BI 工具高频触发。教科书级"伪装成轻访问器"。 +- **修复方向**：建表/schema-cache 时捕获 comment 随表对象缓存，或该路径按需惰性取。 + +### [ ] PERF-06 — C3：vended-credentials 每文件重建 StorageProperties +- **病灶**：`buildRange:1105 normalizeUri` / `convertDelete:1157 → DefaultConnectorContext.normalizeStorageUri:392-409 → buildVendedStorageMap:225-242 → StorageProperties.createAll`（遍历所有 provider + 建 hadoop `Configuration` + 逐 key set），**每 data file 和每 delete file 各一次**，而 vended token 整个 scan 内不变。50k 文件 ≈ 数十秒纯 FE CPU。门槛：REST + `iceberg.rest.vended-credentials-enabled=true`（MOR 加倍）。 +- **修复方向**：token→typed-map 推导按 scan 提升 / 在 `DefaultConnectorContext` 内做单条目 memo（token 恒等键）。 + +### [ ] PERF-07 — C20：写路径 3~5 次 load 同表 +- **病灶**：`PhysicalIcebergMergeSink.getRequirePhysicalProperties:161→198`、`PhysicalPlanTranslator.visitPhysicalConnectorTableSink:675,703`、`PluginDrivenTableSink.bindDataSink:175 → IcebergWritePlanProvider.resolveTable:689-702 / beginWrite`（内含 tableExists×2 + 无条件 refresh）。每条 DML +3~5 次串行远程往返。 +- **修复方向**：语句级 resolve 一次传递；exists 从 load 结果推导。 + +--- + +## P2 — CPU/payload/维护路径，影响门槛高但模式典型 + +### [ ] PERF-08 — 维护路径逐单位重扫/无去重（C19 C21） ·改动小、可先行插队 +> 覆盖两条独立维护命令，同一"逐单位重扫、零去重"模式。同一任务，实现上可各自独立 commit。 +- **C19 rewrite_data_files**：`ConnectorRewriteDriver.run STEP3:143` 每个 rewrite group 调一次 `registerRewriteSourceFiles` → 每次一遍 `useSnapshot(...).planFiles():379-383`。G 个 group = G+1 次整表扫描，G~50-200 时分钟级。修：**union 所有 group 一次注册**（SPI 本来就收 `Set`）。 +- **C21 expire_snapshots**：`IcebergExpireSnapshotsAction.buildDeleteFileContentMap:271-293` 对**每个** snapshot 读其全部 delete manifest、无 visited-path 去重 —— 相邻 snapshot manifest 大量重叠，S×M 串行远程读。修：按 `ManifestFile.path()` 去重坍缩为 O(distinct)。 + +### [ ] PERF-09 — C5：streaming pump 逐 split 重建 backend 候选集（**fe-core 框架层**） +- **病灶**：`startStreamingSplit:1638-1642` 逐 split `addToQueue` → `SplitAssignment` 每 split 重建 backend 候选集（`FederationBackendPolicy.computeScanRangeAssignment:225-235` 全量 backend 拷贝 + shuffle + multimap）+ `synchronized` 往返。10⁵~10⁶ split × ~100 BE ≈ 10⁷~10⁸ 冗余操作。 +- **修复方向**：pump 侧微批（64~256 个/批）。 +- **约束**：⚠ 改的是 **fe-core 通用框架**（惠及所有连接器，非 source-specific，允许改）；但属共享热路径，须证 **byte + cost 对所有连接器双不变**（对齐现有"共享 MVCC 方法须双不变"纪律）。 + +### [ ] PERF-10 — C8：WHERE conjunct 转换 5~6 次/查询 +- **病灶**：同组 conjunct 在 `buildRemainingFilter`（×3 处）+ `buildScan:974` + `getScanNodeProperties:1428`（EXPLAIN 专用序列化，**非 EXPLAIN 也跑**）被转换 5~6 次。单次微秒~毫秒级，但与簇1 同源叠加。 +- **修复方向**：node 字段 memo，与 `cachedPropertiesResult` 同点失效。 +- **依赖**：与 PERF-01 同源，宜在 01 之后顺手。 + +### [ ] PERF-11 — 簇5：per-split 不变量 + payload 放大（C12 C13 C14 C15） +> 同一 `buildRange`/`populateRangeParams` 区域的 per-split 重复计算与 payload 放大，作为一个任务批量处理（可拆多个 commit）。 +- **C12**：一个 data file 被 `TableScanUtil.splitFiles` 切成 k 个 byte-slice 后，`buildRange:1045` 对每个 slice 重算 partition JSON（Jackson 序列化 + 时区格式化）、identity map、delete 转换 —— (specId, PartitionData) 级不变量。100k split ≈ 0.5~2s CPU。修：按 (specId, PartitionData) memo。 +- **C13**：v3 rewritable-delete stash：同一 delete 列表 plan 期 `rewritableDeleteDescs:302-313` 转一次 thrift，`populateRangeParams` 再转一次；每 slice 重复 put 相同 supply。修：转一次复用、per-file 而非 per-slice。 +- **C14**：通用节点 per-split 重复 `LocationPath.of`（URLEncoder+URI.create）、重建 columns-from-path 却被 `IcebergScanRange.populateRangeParams:435-437` unset 丢弃；`resolveScanProvider` 每 split 经 `getFileCompressType` 反复解析。修：hoist 不变量 / 删造完即弃分支。⚠ 涉 **fe-core 通用节点**——保持 connector-agnostic，勿加 source-specific 分支。 +- **C15**（payload 放大）：同一 data file 的完整 delete 列表 + partition JSON 复制进**每个** byte-slice 的 `TFileRangeDesc`；共享 delete file 逐 data file 重复。大 MOR 扫描计划体积多出 MB 级（FE 构建 + BE 解析双向付费）。修：per-file 共享引用，不逐 slice 复制 payload。 + +--- + +## 不立项（记录） + +- **R1（驳回）**：字典新鲜度 poll 每 5s 重建分区视图 —— 重操作属实但乘数不成立（`CreateDictionaryInfo.validateAndSet:164` 对外表强转 `catalog.Table` ⇒ CREATE DICTIONARY 在校验期即 ClassCastException，poll 路径不可达）。详见报告 §3。 +- **旁获（与本审计无关，另行处理）**：上述强转本身是功能缺口/潜在 bug（外表字典完全不可用）。**不在本任务空间**，如需修单开任务。 From c6ec41d97aa8f820f5713598a93a78d659cf54e5 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 17 Jul 2026 23:31:42 +0800 Subject: [PATCH 03/47] [doc](catalog) fe-connector-iceberg PERF-01: finalize table-cache design Re-verify + adversarial red-team + multi-round design for PERF-01 (the 3~7x remote loadTable amplification per query). No product code touched. Key findings folded into designs/FIX-PERF-01-table-memo-design.md: - loadTable is snapshot-agnostic -> memo key is TableIdentifier only, not (id, snapshotId) as the audit draft assumed. - Legacy IcebergMetadataCache (pre-#60937) already cached the Table object cross-query, so "behavior parity" means cross-query caching, not the audit's per-planning-pass framing. - Red-team refuted Part B (convertPredicate invalidation narrowing): in doFinalize, convertPredicate runs before any property build, so clearing the (still-null) caches is a no-op. The real duplication is the dual build path (getSplits + properties), which Part A covers. Part B dropped. - Red-team BLOCKER: the cross-query disable gate must be isUserSessionEnabled() || restVendedCredentialsEnabled() (vended tokens expire ~60min and would 403 mid-scan on a 24h-TTL cache hit). Finalized design (aligned with Trino, user-approved): - Fat handle: transient resolvedTable on IcebergTableHandle -> one Table instance per query, auto-reclaimed with the plan, zero per-query state on the connector (the Doris analog of Trino's per-transaction handle). - Cross-query IcebergTableCache on the long-lived IcebergConnector, consumed in the read helper (not a catalog-seam decorator, so DDL stays isolated), gated off for session=user / vended-credentials. Next: implement PERF-01 (TDD, RecordingIcebergCatalogOps count gate first). Co-Authored-By: Claude Opus 4.8 (1M context) --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 63 ++++++------ .../designs/FIX-PERF-01-table-memo-design.md | 95 +++++++++++++++++++ plan-doc/perf-hotpath-iceberg/progress.md | 10 ++ plan-doc/perf-hotpath-iceberg/tasklist.md | 2 +- 4 files changed, 140 insertions(+), 30 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-01-table-memo-design.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index 004f3020f4de49..45faaa6afe0ee8 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -1,50 +1,55 @@ # 🤝 Session Handoff — fe-connector-iceberg 热路径重操作修复 -> **滚动文档**：每次 session 结束**覆盖式更新**，只保留下一个 session 必须的上下文；完成明细进 `git log` + `designs/` + `progress.md`，别堆这里。 -> **范围** = 逐一修复审计报告 `../reviews/perf-audit-fe-connector-iceberg-2026-07-17.md` 的 23 条发现（C1–C23 → **一簇一任务** 11 个 `PERF-NN` 任务，见 [`tasklist.md`](./tasklist.md)）。 +> **滚动文档**：每次 session 结束**覆盖式更新**，只保留下一个 session 必须的上下文；完成明细进 `git log` + `designs/` + `progress.md`。 +> **范围** = 逐一修复审计报告 `../reviews/perf-audit-fe-connector-iceberg-2026-07-17.md` 的 23 条发现 → **11 个 `PERF-NN` 任务**，见 [`tasklist.md`](./tasklist.md)。 > 开场必读顺序、单项立项流程、约束铁律：[`README.md`](./README.md)。 --- -# 🆕 下一个 session = **启动 PERF-01（簇1，收益最大）** +# 🆕 下一个 session = **实现 PERF-01（设计已定稿，进入编码）** -## 现状（session 0，2026-07-17） +## 现状（session 1，2026-07-17） -- 本任务空间**刚建好**：`README.md` / `tasklist.md` / `HANDOFF.md` / `progress.md` / `designs/` 就位。 -- 粒度已定：**一簇一任务**（用户 2026-07-17 拍板）—— 23 发现归并为 **11 个任务**。 -- **零任务已实现** —— 全部 11 项 `⏳ 待启动`。 -- 审计报告是**待 review 的结论草案**（报告 §5 自称"供 review 讨论，非结论"）。因此**不照单硬修**：每项第一步是**按 findings.json 的调用链复核行号 + 乘数**（README §单项立项流程 step 2），复核站得住再设计实现。 +- PERF-01 **复核 + 红队 + 设计全部完成并定稿**，写入 [`designs/FIX-PERF-01-table-memo-design.md`](./designs/FIX-PERF-01-table-memo-design.md)。**未动任何产品代码**。 +- 设计经用户多轮对齐 Trino 拍板，**最终形态**： + - **① 胖 handle**：`IcebergTableHandle` 加 `transient Table resolvedTable`（不序列化）+ getter/setter；读表统一"胖 handle 优先"；`withSnapshot/withRewriteFileScope/withTopnLazyMaterialize`（`:182/198/207`）**携带前行**；sys 表 handle 不用。→ 查询内单实例、随查询计划自动回收、**连接器侧零每查询累积**。 + - **② 跨查询 `IcebergTableCache`**（新类，仿 `IcebergLatestSnapshotCache`，值=raw Table）：挂长生命周期 `IcebergConnector`，传进每个 fresh metadata + provider，**在读 helper 里消费**（非 catalog-seam 装饰器 → DDL 天然隔离）。**gate=`isUserSessionEnabled() || restVendedCredentialsEnabled()` 关闭**（红队 BLOCKER：vended token 过期会 403）。 + - **Part B（convertPredicate 收窄）已删**（红队证伪为 no-op）。 +- 度量守门：`RecordingIcebergCatalogOps` 断言规划期对同一表 `loadTable` 远端=1；全查询 1（跨查询开）/≤2（关）。 -## 下一个 session 第一件事（精确到动作） +## 下一个 session 第一件事（TDD，精确到动作） -1. 向用户复述并确认起点："上次建好了 iceberg 热路径修复任务空间，下一步启动 **PERF-01（簇1，per-planning-pass Table memo）**，对吗？"（或用户另指定某项，如想先做改动小的 **PERF-08**＝维护路径 C19/C21）。 -2. 确认后，**先复核 PERF-01**（design subagent）： - - `grep` 复核簇1 的 7 个 loadTable 调用点行号（报告簇1 表 / findings.json 里 C1 C4 C6 C10 C16 的 `heavy_op` + `multiplicity` 字段）：`IcebergCatalogOps.java` 的 `loadTable`（报告记 :340）、`IcebergConnectorMetadata.getColumnHandles`（:587）、`IcebergScanPlanProvider.getScanNodeProperties`（:1300/1311）+ `resolveTable`（:1981-1993）、`convertPredicate`（:795-798 无条件清 `cachedPropertiesResult`）、`streamingSplitEstimate`（:410）、`planScanInternal`（:562）。 - - 确认无 `CachingCatalog`（全仓 grep）、`IcebergLatestSnapshotCache` 确只存 `(snapshotId,schemaId)`。 - - 确认 `beginQuerySnapshot` 的 pin 可作 `(TableIdentifier, snapshotId)` memo key。 -3. 复核站得住 → 写 `designs/FIX-PERF-01-table-memo-design.md`（含**度量方案**：用调用计数证明 loadTable 从 N→1），设计红队，再实现。 +1. 先读 `designs/FIX-PERF-01-table-memo-design.md`（§3 设计 / §4 实现计划 / §5 风险 / §6 度量守门）——这是权威 spec。 +2. **先写测试（TDD）**：`RecordingIcebergCatalogOps`（test 已存在）计数守门——规划期对同一表 `loadTable` 远端次数 = 1（修前基线先记录）。 +3. 按 §4 实现计划 5 步落地（Commit 1，一个 commit）： + - `IcebergTableHandle`：transient `resolvedTable` + getter/setter + 3 个 `with*` 携带。 + - 新增 `IcebergTableCache.java`（仿 `IcebergLatestSnapshotCache`，值 `Table`）。 + - `IcebergConnector`：构造 `tableCache`、传入 metadata/provider、`invalidate*` 三钩子（`:523-553`）、gate。 + - `IcebergConnectorMetadata.loadTable(handle):540` 改"胖 handle 优先→跨查询缓存→remote→setResolvedTable"；`getColumnHandles`/`getTableStatistics`/`resolveTimeTravel`/`getMvccPartitionView`/`listPartitions`/`beginQuerySnapshot` 经它；sys 分支不动。 + - `IcebergScanPlanProvider.resolveTable:1981` 改"胖 handle 优先"+ per-call `wrapTableForScan`；构造加 `tableCache` 参。 +4. 验证（parity + 减负）→ 独立 commit `[perf](catalog) fe-connector-iceberg: (PERF-01)` → 写 `designs/FIX-PERF-01-...-summary.md` → 更新 tasklist/progress/本 HANDOFF。 ## ⚠️ 关键认知（别重踩） -- **PERF-01 必须最先做**：它的 `convertPredicate` 失效收窄是 PERF-03（簇2 消第二次计算）和 PERF-10（C8）的前置；它立住的 `(table, snapshotId)` memo 模式被 PERF-02/03 复用。顺序错了要返工。 -- **这是性能修复 = 行为必须不变**：验收闸门是「相关 UT/e2e 全绿（parity）+ 证明减负（调用计数/时延）」，不是新增功能。每项尽量补一个**调用计数守门**（如断言规划期 loadTable=1），否则回归无法自动发现。 -- **memo 放连接器侧，别碰 fe-core 源**（fe-core 只出不进铁律）。PERF-09（C5 微批）、PERF-11 内 C14（通用节点）虽改 fe-core 通用层（允许，因非 source-specific），但须证跨连接器 byte+cost 双不变、禁按源名分支。 -- **审计草案可能被复核推翻**：若某条复核后乘数/路径不成立，tasklist 标 🔬 + 记 progress，**不硬凑修复**。 +- **gate 两半都要**：`isUserSessionEnabled() || restVendedCredentialsEnabled()`（后者独立于前者，红队 Attack 2 = BLOCKER）。关的只是跨查询层；胖 handle 层不 gate（查询内 token 新鲜）。 +- **胖 handle 存 raw Table**；`resolveTable` 命中后仍 per-call `wrapTableForScan`（Kerberos FileIO 不冻进缓存）。 +- **DDL/写/procedure 不碰 resolvedTable、不经读 helper**（走裸 ops 拿 fresh base）。 +- **memo 放连接器/handle 侧，别碰 fe-core 源**（fe-core 只出不进铁律）。`IcebergTableCache` 挂 `IcebergConnector`，`resolvedTable` 挂 `IcebergTableHandle`——都在连接器侧。 +- 携带 transient 的原因：`getColumnHandles` 在 pin 前 set，`pinMvccSnapshot` 用 `withSnapshot` 换 handle，不携带则 pin 后重解析。 --- -# 🧰 构建/验证坑（承自主线 HANDOFF，本空间直接复用，别再踩） +# 🧰 构建/验证坑（承自主线 HANDOFF，直接复用） -1. **maven build cache 会静默跳过 surefire** —— 日志 `Skipping plugin execution (cached): surefire:test` 时 **BUILD SUCCESS 是空的**（报告是陈旧文件）。**所有测试必须加 `-Dmaven.build.cache.enabled=false`**。 -2. **`mvn ... | tail` 后的 `$?` 是 `tail` 的**，不是 maven 的 —— 重定向到文件再取 `$?`，或读 `BUILD SUCCESS`/`BUILD FAILURE` 行。 -3. **maven 用绝对 `-f /fe/pom.xml`**（cwd 跨调用持久，`cd` 会破相对路径 / 触发权限弹窗）。 -4. **`regression-test/conf/regression-conf.groovy` 工作区本就是脏的**（session 开始前即 `M`）—— **别顺手 `git add -A`**，按需精确 add。 -5. **并发 session 共享同一 worktree** —— 动码前探测并发活动（`git log/status` + maven 进程 + 近 90s mtime）；`pgrep maven` 可能查到 1 天前的僵尸 until-loop，**看 `etime` 再判**，别误判成活跃 session 无谓停手。小步快提交缩短被 amend 卷走窗口。 -6. **本 worktree 的 `.git` 是文件不是目录**（linked worktree）—— rebase/冲突脚本禁硬编码 `.git/rebase-merge`，一律 `git rev-parse --git-path`。 +1. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 被静默跳过，BUILD SUCCESS 是陈旧文件）。 +2. **`mvn ... | tail` 后的 `$?` 是 `tail` 的** —— 读 `BUILD SUCCESS`/`BUILD FAILURE` 行。 +3. **maven 用绝对 `-f /fe/pom.xml`**（cwd 跨调用持久，`cd` 破相对路径）。 +4. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。 +5. **并发 session 共享同一 worktree** —— 动码前探测（`git log/status` + `pgrep maven` 看 `etime`）。 +6. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 --- -# 🗂 开放问题 / 待用户裁决（不是 TODO，是需要拍板的） +# 🗂 开放问题 -- **审计整体是否已被用户接受为立项依据**：目前按"每项立项前自行复核行号/乘数"处理。若用户已整体签字，可省去部分复核力度（但行号复核仍建议保留）。 -- ~~任务粒度~~：已定 **一簇一任务**（2026-07-17），无遗留。 +- 无。设计定稿，进入实现。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-01-table-memo-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-01-table-memo-design.md new file mode 100644 index 00000000000000..52ea6b46d1a6c0 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-01-table-memo-design.md @@ -0,0 +1,95 @@ +# FIX-PERF-01 — 跨查询 Table 缓存（恢复 legacy 表缓存）+ 属性失效收窄 + +> 任务：消除"一次 iceberg 查询规划里同一张表被远程 loadTable 3~7 次"的重操作放大。 +> 覆盖审计发现 C1 C4 C6 C10 C16（簇1）。 +> 基线：HEAD `b342570d6ca`，分支 `catalog-spi-review-16`。行号信 grep 不信本文档。 +> 决策已定（用户 2026-07-17）：**跨查询缓存（与出厂 legacy parity）+ 自建缓存复用现有 `MetaCacheEntry` 基建（A2，= legacy `IcebergMetadataCache` 结构）**。 + +--- + +## 1. Problem + +一条带 `WHERE` 、扫分区表的普通 `SELECT`，在规划期把同一张表远程 `loadTable`（= metastore RPC + 读 metadata.json）**7 次**： + +| 次数 | 调用点 | file:line | +|---|---|---| +| ×3 | `getColumnHandles` → `loadTable`（属性第1次 build + 属性第2次 build + getSplits） | `IcebergConnectorMetadata.java:587` | +| ×3 | `resolveTable` → `ops.loadTable`（getScanNodeProperties 第1次 + 第2次 + planScan） | `IcebergScanPlanProvider.java:1300 / 562 / 1988` | +| ×1 | `streamingSplitEstimate` → `resolveTable` | `IcebergScanPlanProvider.java:410` | + +**真正的重复来自 dual-build**（红队 2026-07-17 证伪审计的"convertPredicate 触发第二次重算"）：`doFinalize`（`FileQueryScanNode:248-253`）先 `convertPredicate()`(:252) 后 `createScanRangeLocations()`(:253)；属性缓存 `cachedPropertiesResult` 在 `createScanRangeLocations` 内首次 build（`PluginDrivenScanNode.getOrLoadPropertiesResult:1776`）→ **convertPredicate 跑时三字段皆 null，清空是 no-op**。重复实为**两条独立 build 路径各解析一次表**：getSplits 路径（`buildColumnHandles:1202` + planScan 的 resolveTable）与属性路径（`buildColumnHandles:1780` + `getScanNodeProperties:1300` 的 resolveTable），外加 `streamingSplitEstimate:410`、分析期 `getMvccPartitionView:1453`/`listPartitions:1507`、CBO `getTableStatistics:624`。 + +## 2. Root Cause（含复核对审计草案的修正） + +- **根因**：SPI 迁移把 legacy `IcebergMetadataCache`（`LoadingCache tableCache`，`getIcebergTable()` 返回缓存 Table，快照缓存从缓存 Table 派生 —— 见 #60937 前的该文件 `:57/:102-108/:114`）拆开后，**只移植了"快照 id"那半**（`IcebergLatestSnapshotCache` 只存 `(snapshotId, schemaId)` 两个 long，`:57-64`），**丢了"缓存 Table 对象"那半**。于是各 SPI 入口只能各自 `loadTable`。全仓无 `CachingCatalog`（只 paimon 有）。 +- **修正1（缓存键）**：`catalog.loadTable()` **快照无关**（永远返回最新元数据；查询要读的快照 id 在下游 `useSnapshot()` 才应用）。故 memo 键**只需 `TableIdentifier`**，审计草案的 `(id, snapshotId)` 是过度限定；一张表一条语句里的所有时间旅行/分支引用共用一个 Table 对象。也消除了"最早入口 `getColumnHandles` 在快照 pin 之前"的疑虑。 +- **修正2（parity 基准）**：legacy 本就**跨查询**缓存 Table（带 TTL、REFRESH 失效）。因此"行为不变（与出厂 legacy parity）"= 跨查询缓存；审计草案的"per-planning-pass"反而比 legacy 多加载、且偏离出厂行为。paimon 亦跨查询（SDK CachingCatalog）。 +- **修正3（Part B 证伪）**：审计与我早先复核以为"带 WHERE 时 `convertPredicate` 清空属性缓存 ⇒ 第二次重算"。**红队用控制流证伪**：convertPredicate 先于任何 build 跑（见 §1），清空 no-op，无第二次重算可消。故 **Part B 删除**（详见 §3 Part B）。真正的重复是 dual-build，Part A 覆盖。 + +## 3. Design + +### Part A（定稿）：胖 handle（查询内单实例）+ 跨查询 `IcebergTableCache`（挂 `IcebergConnector`） + +**背景（实例数,已核实)**：只有 `IcebergConnector` 及其内 SDK `Catalog` 是"每 catalog 1 个、长期存活"；`IcebergConnectorMetadata`/`IcebergScanPlanProvider`/`IcebergCatalogOps` 都"每次调用 new、一查询多个、用完即弃"。故跨调用共享的状态只能挂 **①长生命周期 `IcebergConnector`** 或 **②贯穿一条查询的 `handle`**。定稿两层各取其一： + +**第一层 —— 胖 handle（查询内单实例,一致性由构造保证,随查询自动回收,连接器侧零累积)** +给 `IcebergTableHandle` 加 `private transient Table resolvedTable`（不序列化）+ getter/setter。读表统一走"胖 handle 优先":有则直接返回**同一实例**,无则解析后 `setResolvedTable` 挂上。`withSnapshot/withRewriteFileScope/withTopnLazyMaterialize`（`:182/198/207`）拷贝时**携带前行**（iceberg 分支/标签/时间旅行是**同一个 Table** + 下游 `useSnapshot/useRef`；`getColumnHandles` 在 pin 前 set、`pinMvccSnapshot` 用 `withSnapshot` 换 handle,故必须携带否则 pin 后重解析）。sys 表 handle **不**用（走 `loadSysTable`）。存 **raw** Table；`resolveTable` 命中后 per-call `wrapTableForScan`。 + +**第二层 —— 跨查询 `IcebergTableCache`（挂 `IcebergConnector`,legacy parity + 连续查询命中,作为胖 handle 首次 miss 的 backing）** +新增 `IcebergTableCache`（值=raw Table），复用 `IcebergLatestSnapshotCache` 同款 `MetaCacheEntry` 基建（TTL=`meta.cache.iceberg.table.ttl-second` 24h，REFRESH 失效）；挂 `IcebergConnector`，像 `latestSnapshotCache` 一样传进每个 fresh metadata **和** provider。**在读 helper 里消费**（非 catalog-seam 装饰器 → DDL 天然隔离）。对 `isUserSessionEnabled() || restVendedCredentialsEnabled()` **关闭**（§5.1）。 + +**统一读 helper**（`Metadata.loadTable(handle):540` 与 `ScanProvider.resolveTable:1981` 同规则）： +``` +Table resolve(handle): + if handle.resolvedTable != null: return handle.resolvedTable // 胖 handle → 同一实例 + t = executeAuthenticated( tableCache!=null // 跨查询层(gated→null) + ? tableCache.getOrLoad(id, () -> catalogOps.loadTable(db,tbl)) + : catalogOps.loadTable(db,tbl) ) + handle.setResolvedTable(t); return t // provider 侧返回 wrapTableForScan(t) +``` + +**去重计数（诚实）**：规划期所有读共享 `currentHandle` 血缘 → **规划期 1 次**；分析期同理 1 次；分析↔规划不同血缘,靠跨查询层桥接 → 跨查询开 **全查询 1 次远端**,关(session=user/vended) **≤2 次**（仍远小于 7~10）。 + +**失效**：`IcebergConnector.invalidate{Table,Db,All}`（`:523-553`）各加 `tableCache.invalidate*`（与 `latestSnapshotCache` 并列）；handle 的 resolvedTable 随查询回收,无需失效。 +**DDL/写隔离**：DDL/写/procedure 不碰 handle 的 resolvedTable、不经读 helper → 走裸 ops 拿 fresh base。 +**鉴权/线程**：读 helper 已在 `executeAuthenticated` 内;命中零远端;miss 时 `delegate.loadTable` 在该 scope 跑 → auth 正确。缓存 raw Table 跨并发共享安全（红队 Attack 4 确认）;Kerberos FileIO 由 `wrapTableForScan` per-call 施加。 + +### Part B（副线）：~~属性失效收窄~~ —— **已删除（红队证伪）** + +原设想：把 `PluginDrivenScanNode.convertPredicate:796-798` 的失效收进 `if (result.isPresent())`。**红队用控制流证伪**：`doFinalize` 先 convertPredicate、后 createScanRangeLocations（首次 build），故 convertPredicate 时 `cachedPropertiesResult/scanNodeProperties/filteredToOriginalIndex` 皆 null → 清空是 **no-op**，**消不掉任何真实重算**；且它是"若未来引入 finalize 前的属性 build 就会读到陈旧缓存"的**潜在隐患**（`filteredToOriginalIndex` 被 `pruneConjunctsFromNodeProperties:1739` 读，`PUSHDOWN_PREDICATES_PROP:1428` 依赖 conjunct）。→ **本任务不做 Part B**；独立 WHERE-conjunct 任务（C8）如仍需碰此点，须先用 `RecordingIcebergCatalogOps` 栈追证明存在"finalize 前的真实第一次 build"再动，否则同样删除。 + +## 4. Implementation Plan（最小改动，串行） + +**Commit 1（Part A，一个 commit，iceberg 自包含）**： +1. `IcebergTableHandle`：加 `transient Table resolvedTable` + getter/setter；`withSnapshot/withRewriteFileScope/withTopnLazyMaterialize`（`:182/198/207`）携带前行。 +2. 新增 `IcebergTableCache.java`（跨查询层，仿 `IcebergLatestSnapshotCache`，值 `Table`；`getOrLoad/invalidate/invalidateDb/invalidateAll/isEnabled/size`）。 +3. `IcebergConnector`：构造 `tableCache`；`getMetadata`/`getScanPlanProvider` 传入；`invalidate*` 三钩子加跨查询失效；gate=`isUserSessionEnabled() || restVendedCredentialsEnabled()` 关跨查询层（§5.1）。 +4. `IcebergConnectorMetadata`：`loadTable(handle):540` 改"胖 handle 优先 → 跨查询缓存 → remote → setResolvedTable"；`getColumnHandles`/`getTableStatistics`/`resolveTimeTravel`/`getMvccPartitionView`/`listPartitions`/`beginQuerySnapshot` 经它。sys 分支不动。 +5. `IcebergScanPlanProvider`：生产构造器加 `tableCache` 参；`resolveTable:1981` 改"胖 handle 优先"+ per-call `wrapTableForScan`。 + +**Part B 不做**（§3 Part B 已删）。 + +## 5. Risks & Open Questions + +1. **凭证时效/隔离（正确性红线，BLOCKER，红队 Attack 2 加强）**：第二层跨查询缓存必须对 **`isUserSessionEnabled() || restVendedCredentialsEnabled()`** 关闭（**两半都要**，二者独立）： + - session=user（`IcebergConnector:675`）：per-user ops，raw Table 携带 per-user delegated FileIO → 跨用户共享 = 凭证泄漏。 + - plain-REST + `iceberg.rest.vended-credentials-enabled=true`（`IcebergScanPlanProvider:1243`，**不**依赖 session=user）：raw Table 的 FileIO 携带 ~60min 过期的 vended token；代码自注（`:1256`）"credentials 靠每查询重载表保持新鲜"。24h TTL 内命中 → BE 拿**过期 token → 403 中途失败**。 + 关闭的只是第二层；**第一层 transient（查询内单实例）不受影响**（同一查询内 token 新鲜）。快照 id 缓存（用户无关）不变。补 UT：plain-REST-vended（无 session=user）断言 Table 不跨查询共享。 +2. **DDL 隔离**：Table 缓存只挂 scan 读路径（`getColumnHandles`/`beginQuerySnapshot`/`resolveTable`）；DDL 的 `loadTable().updateSchema().commit()`（`IcebergCatalogOps:397-478`）**不走缓存**，用裸 `catalogOps` → 提交拿 fresh base，无陈旧提交风险。 +3. **schema-only ALTER 可见性**：跨查询缓存下，"只改 schema 不产生新快照"的外部 ALTER 要等 TTL/REFRESH 才可见 —— **= legacy 行为**（legacy 快照/schema 亦从缓存 Table 派生），且当前快照缓存已用同样 24h TTL 冻结 → 非新增 staleness。 +4. **两套缓存 TTL 不同步**：Table 缓存与 id 缓存独立淘汰，可能 id 命中但 Table miss（或反之）→ 重载 Table 拿 fresh，id 稍旧 —— 仍在"快照 TTL 内稳定"语义内，可接受。 +5. **胖 handle 跨相不桥接**：分析期与规划期是不同 handle 血缘,transient 不跨相 → 分析↔规划的一致性靠跨查询层桥接（开时同一对象;关时极罕见两阶段各自 load 出不同代,但快照 pin 已钉、数据一致,仅 schema 极罕见偏移）。可接受(= 任何缓存在该边界的表现)。 +6. **内存**：连接器侧**零每查询累积**（Table 引用挂各查询 handle,随查询 GC）;重 Table 对象在跨查询 `IcebergTableCache` 共享,受其 maxSize 兜（legacy 亦缓存 Table）。 + +## 6. Test Plan / 度量守门 + +- **度量守门（核心）**：用现成 `RecordingIcebergCatalogOps` 断言 `loadTable` 远端次数：**规划期对同一表恰好 1 次**（胖 handle 血缘内去重,确定性,不随分区/CBO/时间旅行变）；**全查询**跨查询层开=1（分析填、规划命中）、关(session=user/vended)=≤2（分析 1 + 规划 1）。修前基线记录以证从 7~10 大降。 +- **Parity UT**：iceberg 连接器现有 UT 全绿（`IcebergConnectorMetadataTest`/`IcebergScanPlanProviderTest`/`IcebergLatestSnapshotCacheTest` 等）。 +- **失效 UT**：REFRESH TABLE/DB/CATALOG 后跨查询层清空、下次 load 走 live（仿 `IcebergLatestSnapshotCacheTest`）。 +- **凭证隔离 UT**：`isUserSessionEnabled() || restVendedCredentialsEnabled()` 下跨查询层关闭（第二次查询仍走 real load，不复用上次的 Table/凭证）；每查询 pin 仍在（查询内 1 次）。 +- **DDL 隔离 UT/e2e**：外部 DDL / REFRESH 后 SELECT 反映变更（TTL/REFRESH 边界 = legacy parity）；DDL 提交拿 fresh base（不经读 helper、不被跨查询缓存毒化）。 +- **build**：`-Dmaven.build.cache.enabled=false`（否则 surefire 静默跳过）。 + +## 7. Parity 论证（为什么"行为不变"） + +跨查询层是**恢复** legacy `IcebergMetadataCache` 跨查询表缓存（SPI 迁移丢失的那半），键/TTL/REFRESH 失效与出厂一致；每查询 pin 只把"同一查询内多次远程取同一表"坍缩为一次（同一 Table 对象），快照 id 语义、schema 可见性、时间旅行取快照全部对齐 legacy；唯一收益是"同信息不再重取"。装饰器只拦外部 `loadTable`，DDL/写路径拿 fresh，行为不变。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index feb2b50ba1709e..83d9bbc8bd3a5c 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -12,3 +12,13 @@ - 任务粒度：**一簇一任务**（用户 2026-07-17 拍板）—— 按审计 §1 总览表行边界，簇级并成一项、未成簇的独立发现各一项，共 **11 个任务**。全部 `⏳ 待启动`，零实现。 - 未动任何产品代码；审计报告仍为待 review 草案，约定每项立项前复核行号/乘数。 - **下一步**：见 HANDOFF —— 启动 PERF-01（簇1，per-planning-pass Table memo）。 + +--- + +## 2026-07-17 — session 1：PERF-01 复核 + 设计定稿（未动产品代码） + +- **复核（subagent）**：确认簇1 loadTable 调用点/无 CachingCatalog/`IcebergLatestSnapshotCache` 只存 `(snapshotId,schemaId)`。修正审计草案两点：①`loadTable` **快照无关** → memo 键只需 `TableIdentifier`（非 `(id,snapshot)`）；②legacy `IcebergMetadataCache`（#60937 前）**本就跨查询缓存 Table 对象**（`LoadingCache<…,IcebergTableCacheValue>`），故"行为不变 = 跨查询缓存"（非审计的 per-planning-pass）。 +- **红队对抗**：**驳回 Part B（convertPredicate 收窄）** —— `doFinalize`（`FileQueryScanNode:252-253`）先 convertPredicate 后 createScanRangeLocations（首次 build），convertPredicate 时属性缓存皆 null，清空是 no-op，消不掉重算；真正的重复是 dual-build（getSplits 路径 + 属性路径），Part A 覆盖。**BLOCKER**：跨查询缓存 gate 必须含 `vended-credentials`（非仅 session=user），否则 24h TTL 内命中过期 token → BE 403。 +- **架构澄清**：`IcebergConnectorMetadata`/`IcebergScanPlanProvider`/`IcebergCatalogOps` 均**每调用 new、每查询多个、用完即弃**；仅 `IcebergConnector` + SDK `Catalog` 长生命周期。→ 跨调用共享只能挂 `IcebergConnector` 或 `handle`。 +- **用户定稿设计（多轮对齐 Trino）**：**胖 handle（`IcebergTableHandle.transient resolvedTable`，查询内单实例、随查询计划自动回收、连接器侧零累积，= Trino per-transaction 胖 handle 的 Doris 对应物）+ 跨查询 `IcebergTableCache`（挂 `IcebergConnector`，仿 `IcebergLatestSnapshotCache`，读 helper 消费非 catalog-seam 装饰器→DDL 天然隔离，gate=`isUserSessionEnabled()||restVendedCredentialsEnabled()`）**。去重计数：规划期 1 次、全查询 1 次（跨查询开）/≤2（关）。全设计 + 度量守门见 `designs/FIX-PERF-01-table-memo-design.md`。 +- **未动任何产品代码**。**下一步**：实现（TDD，先 `RecordingIcebergCatalogOps` 度量守门）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index 51d870daf27788..d67fdcb6dfda09 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -15,7 +15,7 @@ | ID | 优先级 | 覆盖发现 | 主题（一句话） | 依赖 | 状态 | commit | |---|---|---|---|---|---|---| -| PERF-01 | P0 | C1 C4 C6 C10 C16 | 一次规划 3~7 次远程 loadTable → per-planning-pass Table memo + 收窄 convertPredicate 失效 | — | ⏳ | | +| PERF-01 | P0 | C1 C4 C6 C10 C16 | 一次规划 3~7 次远程 loadTable → 胖 handle(查询内单实例)+ 跨查询 IcebergTableCache(挂 Connector);~~convertPredicate 收窄~~已删(红队证伪) | — | 🚧 设计定稿,待实现 | | | PERF-02 | P0 | C7 C22 C23 | 分区视图每查询重扫 PARTITIONS 元数据表 → `(table,snapshotId)` 缓存 + MTMV refresh pin | 与 01 共享快照 pin 机制 | ⏳ | | | PERF-03 | P0 | C2 C11 | #64134 复活：`file_format_type` 兜底走整表 planFiles → memoize / 从枚举反推 | 受益于 01 的失效收窄（消第二次） | ⏳ | | | PERF-04 | P1 | C17 C18 | streaming / COUNT(*) 下推旁路 IcebergManifestCache → 两条旁路接回 | — | ⏳ | | From 1e270b16596d88d302e0830fcadcdde352a247d6 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 00:33:25 +0800 Subject: [PATCH 04/47] [perf](catalog) fe-connector-iceberg: memoize the resolved Table within and across queries (PERF-01) A plain WHERE-filtered iceberg SELECT re-loaded the same table 3~7 times per planning pass: getColumnHandles, getTableStatistics, the scan provider's resolveTable (getScanNodeProperties / planScan / streamingSplitEstimate) and the analysis-phase getMvccPartitionView / listPartitions each issued their own remote loadTable (a metastore RPC + a metadata.json read). The SPI cutover split the legacy IcebergExternalMetaCache and kept only the latest-snapshot pin (IcebergLatestSnapshotCache), dropping the cached-Table half. Restore it in two connector-side layers (fe-core untouched): - Query-scoped fat handle: IcebergTableHandle carries a transient raw Table memo (not serialized, not part of identity), carried forward by withSnapshot / withRewriteFileScope / withTopnLazyMaterialize. Reads resolve the table once and reuse the same instance. PluginDrivenScanNode.currentHandle is per-query, so the memo is GC'd with the plan (zero per-query accumulation) => exactly one remote loadTable per table per planning pass. - Cross-query IcebergTableCache (new; mirrors IcebergLatestSnapshotCache, value = raw Table): 24h TTL, REFRESH-invalidated, backs the fat handle's first miss and restores legacy cross-query caching. Disabled (null) for iceberg.rest.session=user and REST vended-credentials, where a cached raw Table's FileIO carries a per-user / soon-expiring token (cross-user leak / 403 mid-scan); the fat handle stays on there (token fresh within one query). Reads go through resolveTableForRead (fat handle -> cache -> raw loadTable), which throws NoSuchTableException unwrapped so listPartitions / getMvccPartitionView keep their concurrent-drop degradation. The scan provider re-applies wrapTableForScan (Kerberos FileIO) per call, never freezing it into the memo. DDL / write / procedure / sys-table paths bypass the helper (fresh remote base). Part B (convertPredicate narrowing) is intentionally dropped (red-team proved it a no-op). Metric gate: RecordingIcebergCatalogOps asserts one planning pass (getColumnHandles + planScan on a single handle) issues exactly one remote loadTable (was two). Full iceberg module suite: 932 pass / 0 fail / 1 skip; checkstyle clean. Behavior parity: pure de-duplication -- snapshot semantics, schema visibility and time travel are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../connector/iceberg/IcebergConnector.java | 36 +++- .../iceberg/IcebergConnectorMetadata.java | 53 ++++- .../iceberg/IcebergScanPlanProvider.java | 60 +++++- .../connector/iceberg/IcebergTableCache.java | 117 +++++++++++ .../connector/iceberg/IcebergTableHandle.java | 41 +++- .../iceberg/IcebergConnectorCacheTest.java | 73 +++++++ .../iceberg/IcebergScanPlanProviderTest.java | 34 +++- .../iceberg/IcebergTableCacheTest.java | 185 ++++++++++++++++++ .../iceberg/IcebergTableHandleTest.java | 60 ++++++ 9 files changed, 633 insertions(+), 26 deletions(-) create mode 100644 fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableCacheTest.java diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java index 08a1670544a285..ff6567d124eab7 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java @@ -159,6 +159,11 @@ public class IcebergConnector implements Connector { // manifest cache is path-keyed, no-TTL, capacity-bounded; it is consumed only when // meta.cache.iceberg.manifest.enable is set (default off → scan uses the SDK planFiles path). private final IcebergLatestSnapshotCache latestSnapshotCache; + // PERF-01: cross-query cache of the RAW iceberg Table (restores the legacy IcebergExternalMetaCache table + // cache that the SPI cutover dropped). null when the catalog's credentials are query-dependent + // (iceberg.rest.session=user / REST vended-credentials) — see the constructor. The query-scoped fat handle + // (IcebergTableHandle.resolvedTable) is always on and independent of this field. + private final IcebergTableCache tableCache; private final IcebergManifestCache manifestCache = new IcebergManifestCache(); // commit-bridge supply (S4 part 2): per-catalog stash carrying a row-level DML's non-equality delete supply // across the scan->write seam — the scan provider fills it (keyed by queryId), the write provider drains it @@ -187,6 +192,19 @@ public IcebergConnector(Map properties, ConnectorContext context this::pluginAuthenticator); this.latestSnapshotCache = new IcebergLatestSnapshotCache( resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY); + // PERF-01 cross-query RAW-table cache. Disabled (null) when the catalog's credentials are + // query-dependent, because a cached raw Table carries its FileIO's credentials: + // - iceberg.rest.session=user: per-user delegated FileIO -> sharing across users leaks credentials. + // - REST vended-credentials: the FileIO carries a server-vended token that expires within ~an hour; + // iceberg keeps it fresh by reloading the table each query, so a 24h-TTL hit would hand BE an + // expired token (403 mid-scan). Both gates are independent; either one disables this layer. + // The query-scoped fat handle stays on in all cases (its token is fresh within the one query). Same + // TTL/capacity as the snapshot cache (the single meta.cache.iceberg.table.ttl-second knob). + this.tableCache = (isUserSessionEnabled() + || IcebergScanPlanProvider.restVendedCredentialsEnabled(this.properties)) + ? null + : new IcebergTableCache( + resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY); } /** @@ -211,7 +229,7 @@ static long resolveTableCacheTtlSecond(Map properties) { @Override public ConnectorMetadata getMetadata(ConnectorSession session) { return new IcebergConnectorMetadata( - newCatalogBackedOps(session), properties, context, latestSnapshotCache); + newCatalogBackedOps(session), properties, context, latestSnapshotCache, tableCache); } /** @@ -522,6 +540,9 @@ private IcebergCatalogOps newCatalogBackedOps(ConnectorSession session) { @Override public void invalidateTable(String dbName, String tableName) { latestSnapshotCache.invalidate(TableIdentifier.of(dbName, tableName)); + if (tableCache != null) { + tableCache.invalidate(TableIdentifier.of(dbName, tableName)); + } } /** @@ -538,6 +559,9 @@ public void invalidateTable(String dbName, String tableName) { @Override public void invalidateDb(String dbName) { latestSnapshotCache.invalidateDb(dbName); + if (tableCache != null) { + tableCache.invalidateDb(dbName); + } } /** @@ -550,6 +574,9 @@ public void invalidateDb(String dbName) { @Override public void invalidateAll() { latestSnapshotCache.invalidateAll(); + if (tableCache != null) { + tableCache.invalidateAll(); + } manifestCache.invalidateAll(); } @@ -579,6 +606,11 @@ IcebergManifestCache manifestCacheForTest() { return manifestCache; } + /** Test-only: the cross-query table cache, or {@code null} when disabled by the credential gate (PERF-01). */ + IcebergTableCache tableCacheForTest() { + return tableCache; + } + @Override public ConnectorScanPlanProvider getScanPlanProvider() { // Mirrors PaimonConnector.getScanPlanProvider: build a fresh provider per call over the lazily-built @@ -588,7 +620,7 @@ public ConnectorScanPlanProvider getScanPlanProvider() { // threaded for parity with the legacy single per-catalog IcebergMetadataOps. return new IcebergScanPlanProvider(properties, this::newCatalogBackedOps, context, manifestCache, - rewritableDeleteStash); + rewritableDeleteStash, tableCache); } @Override diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index 35027f5578f1dd..510d552fef172a 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -144,18 +144,30 @@ public class IcebergConnectorMetadata implements ConnectorMetadata { // IcebergExternalMetaCache parity, mirrors paimon). The 3-arg ctor (direct-construction tests) passes a // DISABLED cache so those reads stay always-live. private final IcebergLatestSnapshotCache latestSnapshotCache; + // PERF-01: cross-query RAW-table cache (null = no cross-query layer). The 3-arg direct-construction tests + // and the credential-gated catalogs pass null; the query-scoped fat handle (IcebergTableHandle) works + // regardless. Consumed only by resolveTableForRead. + private final IcebergTableCache tableCache; public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties, ConnectorContext context) { - this(catalogOps, properties, context, new IcebergLatestSnapshotCache(0L, 1)); + this(catalogOps, properties, context, new IcebergLatestSnapshotCache(0L, 1), null); } + /** Convenience ctor without a cross-query table cache (tableCache null); used by MVCC/statistics tests. */ public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties, ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache) { + this(catalogOps, properties, context, latestSnapshotCache, null); + } + + public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties, + ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache, + IcebergTableCache tableCache) { this.catalogOps = catalogOps; this.properties = properties; this.context = context; this.latestSnapshotCache = latestSnapshotCache; + this.tableCache = tableCache; } // ========== ConnectorSchemaOps ========== @@ -536,16 +548,43 @@ static String buildShowSortClause(Table table) { return "ORDER BY (" + String.join(", ", sortItems) + ")"; } - /** Loads the iceberg {@link Table} through the seam, wrapped in the FE-injected auth context (Kerberos UGI). */ + /** + * Loads the iceberg {@link Table} for {@code handle}, wrapped in the FE-injected auth context (Kerberos + * UGI). Resolution goes through {@link #resolveTableForRead} (fat handle -> cross-query cache -> + * remote), so the many reads sharing one handle in a planning/analysis pass collapse onto a single remote + * {@code loadTable} (PERF-01); a fat-handle hit returns without any remote call. + */ private Table loadTable(IcebergTableHandle handle) { try { - return context.executeAuthenticated( - () -> catalogOps.loadTable(handle.getDbName(), handle.getTableName())); + return context.executeAuthenticated(() -> resolveTableForRead(handle)); } catch (Exception e) { throw new RuntimeException("Failed to load table, error message is:" + e.getMessage(), e); } } + /** + * Resolves the RAW iceberg {@link Table} for {@code handle} with PERF-01's two-layer memo, WITHOUT opening + * an auth scope or wrapping exceptions — callers own both. The query-scoped fat handle comes first (same + * handle instance -> same table); then the cross-query {@link IcebergTableCache} when enabled (else a + * direct remote load); finally the result is memoized back onto the handle. The remote loader's exception + * propagates verbatim (the cache re-throws it unwrapped), so a caller's own {@code NoSuchTableException} + * degradation (the partition-view readers) still fires. Callers needing the auth scope wrap the call in + * {@code executeAuthenticated} (see {@link #loadTable}). NOT used by the sys-table path + * ({@link #loadSysTable}) or the DDL/write path (both take a fresh remote base by design). + */ + private Table resolveTableForRead(IcebergTableHandle handle) { + Table cached = handle.getResolvedTable(); + if (cached != null) { + return cached; + } + Table table = tableCache != null + ? tableCache.getOrLoad(TableIdentifier.of(handle.getDbName(), handle.getTableName()), + () -> catalogOps.loadTable(handle.getDbName(), handle.getTableName())) + : catalogOps.loadTable(handle.getDbName(), handle.getTableName()); + handle.setResolvedTable(table); + return table; + } + /** * Loads the iceberg metadata (system) table for {@code handle} through the seam, wrapped in the * FE-injected auth context (Kerberos UGI). Mirrors legacy @@ -1450,7 +1489,7 @@ public Optional getMvccPartitionView( IcebergTableHandle iceHandle = (IcebergTableHandle) handle; try { return context.executeAuthenticated(() -> { - Table table = catalogOps.loadTable(iceHandle.getDbName(), iceHandle.getTableName()); + Table table = resolveTableForRead(iceHandle); return Optional.of(IcebergPartitionUtils.buildMvccPartitionView(table, iceHandle.getSnapshotId())); }); } catch (Exception e) { @@ -1473,7 +1512,7 @@ public List listPartitionNames(ConnectorSession session, ConnectorTableH return context.executeAuthenticated(() -> { Table table; try { - table = catalogOps.loadTable(iceHandle.getDbName(), iceHandle.getTableName()); + table = resolveTableForRead(iceHandle); } catch (NoSuchTableException e) { LOG.warn("Iceberg table not found while listing partitions: {}.{}", iceHandle.getDbName(), iceHandle.getTableName(), e); @@ -1504,7 +1543,7 @@ public List listPartitions(ConnectorSession session, return context.executeAuthenticated(() -> { Table table; try { - table = catalogOps.loadTable(iceHandle.getDbName(), iceHandle.getTableName()); + table = resolveTableForRead(iceHandle); } catch (NoSuchTableException e) { LOG.warn("Iceberg table not found while listing partitions: {}.{}", iceHandle.getDbName(), iceHandle.getTableName(), e); diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index 70cd39f5f5e076..58a6848a0c3ad3 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -65,6 +65,7 @@ import org.apache.iceberg.TableOperations; import org.apache.iceberg.TableProperties; import org.apache.iceberg.TableScan; +import org.apache.iceberg.catalog.TableIdentifier; import org.apache.iceberg.exceptions.ValidationException; import org.apache.iceberg.expressions.Expression; import org.apache.iceberg.expressions.Expressions; @@ -222,6 +223,11 @@ public class IcebergScanPlanProvider implements ConnectorScanPlanProvider { // (offline tests), in which case stashing is skipped (the supply is exercised only on the post-cutover write // path; pre-flip the provider never runs at all). private final IcebergRewritableDeleteStash rewritableDeleteStash; + // PERF-01: cross-query RAW-table cache shared with the metadata layer, owned by the long-lived + // IcebergConnector and injected via getScanPlanProvider. Nullable — null via the offline-test ctors and + // when the connector's credential gate disables the cross-query layer; when null resolveTable still uses the + // query-scoped fat handle and falls back to a direct remote load. + private final IcebergTableCache tableCache; // FIX-SCAN-METRICS: per-query stash of the iceberg SDK scan diagnostics captured by the attached // IcebergScanProfileReporter during planScan, keyed by session queryId. fe-core drains it @@ -248,8 +254,20 @@ public IcebergScanPlanProvider(Map properties, IcebergCatalogOps ConnectorContext context, IcebergManifestCache manifestCache, IcebergRewritableDeleteStash rewritableDeleteStash) { // Constant resolver: these ctors (offline tests + the pre-session connector paths) bind a single ops that - // ignores the session, so existing behaviour/tests are byte-identical. - this(properties, session -> catalogOps, context, manifestCache, rewritableDeleteStash); + // ignores the session, so existing behaviour/tests are byte-identical. No cross-query cache (tableCache + // null) — the fat handle still dedups within a planning pass. + this(properties, session -> catalogOps, context, manifestCache, rewritableDeleteStash, null); + } + + /** + * Session-aware convenience ctor without a cross-query table cache (tableCache null); used by the offline + * session-routing tests. The query-scoped fat handle still dedups within a planning pass. + */ + public IcebergScanPlanProvider(Map properties, + Function catalogOpsResolver, + ConnectorContext context, IcebergManifestCache manifestCache, + IcebergRewritableDeleteStash rewritableDeleteStash) { + this(properties, catalogOpsResolver, context, manifestCache, rewritableDeleteStash, null); } /** @@ -261,12 +279,13 @@ public IcebergScanPlanProvider(Map properties, IcebergCatalogOps public IcebergScanPlanProvider(Map properties, Function catalogOpsResolver, ConnectorContext context, IcebergManifestCache manifestCache, - IcebergRewritableDeleteStash rewritableDeleteStash) { + IcebergRewritableDeleteStash rewritableDeleteStash, IcebergTableCache tableCache) { this.properties = properties; this.catalogOpsResolver = catalogOpsResolver; this.context = context; this.manifestCache = manifestCache; this.rewritableDeleteStash = rewritableDeleteStash; + this.tableCache = tableCache; } /** @@ -2053,17 +2072,40 @@ static ZoneId resolveSessionZone(ConnectorSession session) { * {@code null} context (offline unit tests / simple-auth) resolves directly. */ private Table resolveTable(ConnectorSession session, IcebergTableHandle handle) { + // Fat handle first (PERF-01): one handle threaded through getColumnHandles + planScan resolves the table + // once. The memo holds the RAW table; wrapTableForScan (the Kerberos doAs FileIO) is re-applied per call + // below so no per-request authenticator is ever frozen into the shared memo/cache. + Table cached = handle.getResolvedTable(); + if (cached != null) { + return wrapTableForScan(cached); + } // Resolve the per-request ops before the auth scope so a session=user fail-closed surfaces verbatim. IcebergCatalogOps ops = catalogOpsResolver.apply(session); + Table raw; if (context == null) { - return ops.loadTable(handle.getDbName(), handle.getTableName()); + raw = loadRawTable(ops, handle); + } else { + try { + raw = context.executeAuthenticated(() -> loadRawTable(ops, handle)); + } catch (Exception e) { + throw new RuntimeException("Failed to load table for scan, error message is:" + e.getMessage(), e); + } } - try { - return wrapTableForScan(context.executeAuthenticated( - () -> ops.loadTable(handle.getDbName(), handle.getTableName()))); - } catch (Exception e) { - throw new RuntimeException("Failed to load table for scan, error message is:" + e.getMessage(), e); + handle.setResolvedTable(raw); + return wrapTableForScan(raw); + } + + /** + * Loads the RAW iceberg table for {@code handle} through the cross-query {@link IcebergTableCache} when + * enabled (the connector disables it for credential-dependent catalogs), else a direct remote + * {@code loadTable}. No wrap and no auth scope here — {@link #resolveTable} owns both. + */ + private Table loadRawTable(IcebergCatalogOps ops, IcebergTableHandle handle) { + if (tableCache != null) { + return tableCache.getOrLoad(TableIdentifier.of(handle.getDbName(), handle.getTableName()), + () -> ops.loadTable(handle.getDbName(), handle.getTableName())); } + return ops.loadTable(handle.getDbName(), handle.getTableName()); } /** diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java new file mode 100644 index 00000000000000..8a6b53ef599787 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableCache.java @@ -0,0 +1,117 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.cache.CacheSpec; +import org.apache.doris.connector.cache.MetaCacheEntry; + +import org.apache.iceberg.Table; +import org.apache.iceberg.catalog.Namespace; +import org.apache.iceberg.catalog.TableIdentifier; + +import java.util.concurrent.ForkJoinPool; +import java.util.function.Supplier; + +/** + * Per-catalog cache of the RAW iceberg {@link Table} object, keyed by {@link TableIdentifier} (db.table) + * (PERF-01). This restores the OTHER half of the legacy {@code IcebergExternalMetaCache} that the SPI cutover + * dropped: {@link IcebergLatestSnapshotCache} kept only the {@code (snapshotId, schemaId)} pin, so every SPI + * read entry ({@code getColumnHandles}, {@code getTableStatistics}, the scan provider's {@code resolveTable}, + * ...) re-loaded the table from the remote catalog (a metastore RPC + a {@code metadata.json} read). This cache + * lets consecutive queries — and the analysis/planning phases of one query, whose handles have distinct memo + * lineages — reuse a single loaded table, exactly as the legacy with-cache catalog did. + * + *

Backing. Reuses the shared {@link MetaCacheEntry} framework identically to + * {@link IcebergLatestSnapshotCache}: a contextual, access-TTL entry whose per-key loader is supplied at + * {@link #getOrLoad}, with manual miss-load on so the loader runs OUTSIDE Caffeine's compute lock + * (single-flight per key) and propagates its exception verbatim (a concurrent-drop + * {@code NoSuchTableException} reaches the caller unwrapped, preserving each read entry's own degradation). + * TTL is {@code meta.cache.iceberg.table.ttl-second} — the same knob that governs the snapshot cache: a + * value {@code <= 0} disables caching (every read goes live), a positive value is Caffeine + * {@code expireAfterAccess} with a {@code maxSize} capacity. Lives on the long-lived per-catalog + * {@link IcebergConnector}; a REFRESH CATALOG rebuilds the connector and thus the cache. + * + *

Values are RAW tables. The scan provider applies {@code wrapTableForScan} (the Kerberos + * {@code doAs} FileIO wrap) per call on the way out, so no per-request authenticator is ever frozen into a + * shared entry. + * + *

Credential isolation. A raw table carries its FileIO's credentials, so this cross-query layer is + * built ONLY when the connector's credentials are query-independent — it is left disabled (the connector + * passes {@code null}) for {@code iceberg.rest.session=user} (per-user delegated FileIO) and REST + * vended-credentials (server-vended tokens expire within the query, and iceberg keeps them fresh by reloading + * the table each query). See {@code IcebergConnector}. + */ +final class IcebergTableCache { + + private final MetaCacheEntry entry; + + IcebergTableCache(long ttlSeconds, int maxSize) { + // Mirror IcebergLatestSnapshotCache: translate the connector's "<= 0 disables" contract to CacheSpec's + // ttl == 0 (disabled) rather than passing a negative value through, which CacheSpec reads as ttl == -1 + // "no expiration (enabled)". + CacheSpec spec = ttlSeconds > 0 + ? CacheSpec.of(true, ttlSeconds, maxSize) + : CacheSpec.of(true, CacheSpec.CACHE_TTL_DISABLE_CACHE, maxSize); + this.entry = new MetaCacheEntry<>("iceberg-table", null, spec, + ForkJoinPool.commonPool(), false, true, 0L, true); + } + + /** Caching is on only when the TTL is positive; ttl-second <= 0 means "always read live". */ + boolean isEnabled() { + return entry.stats().isEffectiveEnabled(); + } + + /** + * Returns the cached table for {@code identifier} if present and unexpired, else runs {@code loader} (the + * live remote {@code loadTable}), caches and returns it. When caching is disabled ({@link #isEnabled()} is + * false) {@code loader} runs every call and nothing is cached. A hit refreshes the entry's expiry + * (access-based). The loader runs OUTSIDE Caffeine's compute lock (single-flight per key) and its exception + * propagates unwrapped. + */ + Table getOrLoad(TableIdentifier identifier, Supplier loader) { + return entry.get(identifier, ignored -> loader.get()); + } + + /** Drops the cached entry for one table so the next read goes live (REFRESH TABLE). */ + void invalidate(TableIdentifier identifier) { + entry.invalidateKey(identifier); + } + + /** + * Drops every cached entry for one database so the next read of any of its tables goes live + * (REFRESH DATABASE / a Doris-issued DROP DATABASE). Entries are keyed by + * {@code TableIdentifier.of(db, table)} (single-level namespace = {@code [db]}), so a db match is + * namespace equality — mirroring {@link IcebergLatestSnapshotCache#invalidateDb}. + */ + void invalidateDb(String dbName) { + Namespace ns = Namespace.of(dbName); + entry.invalidateIf(id -> id.namespace().equals(ns)); + } + + /** Drops all cached entries. */ + void invalidateAll() { + entry.invalidateAll(); + } + + /** Test-only: current number of cached entries (accurate map membership, not Caffeine's estimate). */ + int size() { + int[] count = {0}; + entry.forEach((key, value) -> count[0]++); + return count[0]; + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableHandle.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableHandle.java index 295402a0f66de9..8872f00dc0aaf2 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableHandle.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableHandle.java @@ -20,6 +20,7 @@ import org.apache.doris.connector.api.handle.ConnectorTableHandle; import com.google.common.collect.ImmutableSet; +import org.apache.iceberg.Table; import java.util.Objects; import java.util.Set; @@ -96,6 +97,21 @@ public class IcebergTableHandle implements ConnectorTableHandle { */ private final boolean topnLazyMaterialize; + /** + * Query-scoped resolved {@link Table} memo (PERF-01): the RAW iceberg table this handle resolves to, cached + * so the many read entries that share one handle within a single planning/analysis pass + * ({@code getColumnHandles}, {@code getTableStatistics}, the scan provider's {@code resolveTable}, ...) + * collapse their remote {@code loadTable} RPCs onto one. It is {@code transient} — NOT part of the handle's + * serialized form (BE never sees it) — and NOT part of the handle identity: {@link #equals}/ + * {@link #hashCode}/{@link #toString} deliberately ignore it, since it is a resolution cache, not a + * coordinate. The {@code with*} copies carry it forward (an iceberg branch/tag/time-travel pin is the SAME + * base {@code Table} plus a downstream {@code useSnapshot}/{@code useRef}, and {@code getColumnHandles} + * resolves the table BEFORE the snapshot pin is threaded on), so a pin copy must not force a re-load. A + * deserialized handle (BE side) starts {@code null} and resolves live. The stored table is RAW: the scan + * provider applies {@code wrapTableForScan} (Kerberos FileIO) per call, never freezing it into the memo. + */ + private transient Table resolvedTable; + public IcebergTableHandle(String dbName, String tableName) { this(dbName, tableName, NO_PIN, null, NO_PIN, null, null, false); } @@ -175,6 +191,16 @@ public boolean isTopnLazyMaterialize() { return topnLazyMaterialize; } + /** The query-scoped resolved RAW table memo, or {@code null} if not yet resolved (see {@link #resolvedTable}). */ + public Table getResolvedTable() { + return resolvedTable; + } + + /** Memoizes the resolved RAW table on this handle so later reads sharing it skip the remote load. */ + public void setResolvedTable(Table resolvedTable) { + this.resolvedTable = resolvedTable; + } + /** * Returns a copy of this handle carrying the resolved time-travel pin. Mirrors paimon's * {@code PaimonTableHandle.withScanOptions}/{@code withBranch} but with iceberg's typed carriers. @@ -183,8 +209,13 @@ public IcebergTableHandle withSnapshot(long snapshotId, String ref, long schemaI // sysTableName, rewriteFileScope and topnLazyMaterialize are preserved: threading a resolved // time-travel pin in must not degrade a sys handle (t$snapshots) into a normal data-table handle, // drop a rewrite scope, or drop the lazy-materialization signal. - return new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, + IcebergTableHandle copy = new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, rewriteFileScope, topnLazyMaterialize); + // Carry the resolved-table memo forward: the pin is applied downstream via useSnapshot/useRef on the + // SAME base table, and getColumnHandles resolves the table before applySnapshot copies the handle, so a + // pin copy must not drop the memo and force a re-load (PERF-01). + copy.resolvedTable = resolvedTable; + return copy; } /** @@ -196,8 +227,10 @@ public IcebergTableHandle withSnapshot(long snapshotId, String ref, long schemaI * The other carriers (snapshot/ref/schema/sys) are preserved. */ public IcebergTableHandle withRewriteFileScope(Set rawDataFilePaths) { - return new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, + IcebergTableHandle copy = new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, ImmutableSet.copyOf(rawDataFilePaths), topnLazyMaterialize); + copy.resolvedTable = resolvedTable; + return copy; } /** @@ -205,8 +238,10 @@ public IcebergTableHandle withRewriteFileScope(Set rawDataFilePaths) { * {@link #topnLazyMaterialize}). The other carriers (snapshot/ref/schema/sys/rewriteScope) are preserved. */ public IcebergTableHandle withTopnLazyMaterialize(boolean topnLazyMaterialize) { - return new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, + IcebergTableHandle copy = new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, rewriteFileScope, topnLazyMaterialize); + copy.resolvedTable = resolvedTable; + return copy; } @Override diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java index bc816df8c48e22..0f5c12742f4f47 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java @@ -165,4 +165,77 @@ private static Table tableWithOneManifest() { .commit(); return table; } + + // ==================== PERF-01: cross-query table cache gate + invalidation ==================== + + private static Table fakeTable(String name) { + return new FakeIcebergTable(name, + new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get())), + PartitionSpec.unpartitioned(), "s3://b/" + name, Collections.emptyMap()); + } + + @Test + public void crossQueryTableCacheEnabledForPlainCatalog() { + // A plain catalog (no per-user session, no REST vended credentials) has query-independent credentials, + // so the cross-query RAW-table cache is built and enabled at the default 24h TTL — restoring the legacy + // IcebergExternalMetaCache table cache. MUTATION: leaving it null/disabled for a plain catalog -> the + // 3~7x remote loadTable amplification is not collapsed across queries -> assert below red. + IcebergTableCache cache = + new IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext()).tableCacheForTest(); + Assertions.assertNotNull(cache, "a plain catalog must build the cross-query table cache"); + Assertions.assertTrue(cache.isEnabled(), "the default 24h TTL enables the cache"); + } + + @Test + public void crossQueryTableCacheDisabledForVendedCredentials() { + // REST vended-credentials: the cached raw table's FileIO carries a server-vended token that expires + // within the query (iceberg keeps it fresh by reloading the table each query). A 24h-TTL cross-query hit + // would hand BE an expired token (403 mid-scan), so this layer MUST be off (null); the query-scoped fat + // handle still dedups within one query. MUTATION: building the cache for a vended catalog -> non-null -> red. + Map vended = new HashMap<>(); + vended.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST); + vended.put(IcebergConnectorProperties.REST_VENDED_CREDENTIALS_ENABLED, "true"); + Assertions.assertNull( + new IcebergConnector(vended, new RecordingConnectorContext()).tableCacheForTest(), + "a REST vended-credentials catalog must NOT build the cross-query table cache"); + } + + @Test + public void crossQueryTableCacheDisabledForPerUserSession() { + // iceberg.rest.session=user: the cached raw table carries per-user delegated FileIO, so sharing it + // across users would leak credentials. This layer MUST be off (null) — the fat handle keeps within-query + // dedup. MUTATION: building the cache for a session=user catalog -> tableCacheForTest non-null -> red. + Map session = new HashMap<>(); + session.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST); + session.put(IcebergConnectorProperties.REST_SESSION, IcebergConnectorProperties.SESSION_USER); + Assertions.assertNull( + new IcebergConnector(session, new RecordingConnectorContext()).tableCacheForTest(), + "a per-user session catalog must NOT build the cross-query table cache"); + } + + @Test + public void refreshHooksInvalidateCrossQueryTableCache() { + // The REFRESH hooks must clear the cross-query table cache (else external DDL/writes would stay invisible + // beyond the pin): REFRESH TABLE drops one table, REFRESH DATABASE drops that db's tables, REFRESH + // CATALOG drops everything — mirroring the latest-snapshot cache. MUTATION: an invalidate* hook not + // touching tableCache -> a stale entry survives -> a size assert below red. + IcebergConnector connector = + new IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext()); + IcebergTableCache cache = connector.tableCacheForTest(); + Assertions.assertNotNull(cache); + + cache.getOrLoad(TableIdentifier.of("db1", "t1"), () -> fakeTable("db1.t1")); + cache.getOrLoad(TableIdentifier.of("db1", "t2"), () -> fakeTable("db1.t2")); + cache.getOrLoad(TableIdentifier.of("db2", "t1"), () -> fakeTable("db2.t1")); + Assertions.assertEquals(3, cache.size()); + + connector.invalidateTable("db1", "t1"); + Assertions.assertEquals(2, cache.size(), "REFRESH TABLE drops only that table"); + + connector.invalidateDb("db1"); + Assertions.assertEquals(1, cache.size(), "REFRESH DATABASE drops that db's remaining tables"); + + connector.invalidateAll(); + Assertions.assertEquals(0, cache.size(), "REFRESH CATALOG drops everything"); + } } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index 489ed5c09e5114..af66ca29b6acd0 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -228,6 +228,29 @@ public void planScanResolvesTableInsideAuthContext() { Assertions.assertEquals("db1", ops.lastLoadDb); } + @Test + public void planningPassLoadsSameTableOnceViaFatHandle() { + // PERF-01 metric gate: a single planning pass threads ONE IcebergTableHandle through getColumnHandles + // (metadata) and planScan (provider). The fat-handle memo (transient resolvedTable set by the first + // resolve, reused by the second) collapses BOTH remote reads onto a single loadTable RPC. This is the + // deterministic core claim — one remote loadTable per table per planning pass, independent of the + // cross-query cache (off here: context-less provider, disabled metadata cache). + // MUTATION: dropping the fat handle (each resolve re-loads) -> 2 loadTable log entries -> red. + Table empty = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + RecordingIcebergCatalogOps ops = opsReturning(empty); + IcebergConnectorMetadata metadata = + new IcebergConnectorMetadata(ops, Collections.emptyMap(), new RecordingConnectorContext()); + IcebergScanPlanProvider provider = new IcebergScanPlanProvider(Collections.emptyMap(), ops); + IcebergTableHandle handle = new IcebergTableHandle("db1", "t1"); + + metadata.getColumnHandles(null, handle); + provider.planScan(null, handle, Collections.emptyList(), Optional.empty()); + + long remoteLoads = ops.log.stream().filter("loadTable:db1.t1"::equals).count(); + Assertions.assertEquals(1, remoteLoads, + "fat handle must collapse the metadata + provider reads to one remote loadTable"); + } + // --- T02 split-enumeration + predicate-pushdown tests --- @Test @@ -1760,14 +1783,15 @@ public void planScanNormalizesDataFilePathButKeepsOriginalFilePathRaw() { // --- T05: COUNT(*) pushdown (getCountFromSnapshot + collapse-to-one count range, mirrors paimon) --- - private static final IcebergTableHandle T1 = new IcebergTableHandle("db1", "t1"); - private static List planCount(IcebergScanPlanProvider provider, ConnectorSession session, boolean countPushdown) { // The COUNT-pushdown-aware 7-arg overload the generic PluginDrivenScanNode invokes (limit/ - // requiredPartitions are unused by the iceberg read path). - return provider.planScan(session, T1, Collections.emptyList(), Optional.empty(), - -1L, Collections.emptyList(), countPushdown); + // requiredPartitions are unused by the iceberg read path). A FRESH handle per call mirrors a real + // planning pass: PluginDrivenScanNode.currentHandle is a per-query, per-scan-node handle, so the + // query-scoped fat-handle memo (PERF-01) must never bleed from one test's table to another's (a shared + // static handle would serve the first scenario's cached table to every later one). + return provider.planScan(session, new IcebergTableHandle("db1", "t1"), Collections.emptyList(), + Optional.empty(), -1L, Collections.emptyList(), countPushdown); } @Test diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableCacheTest.java new file mode 100644 index 00000000000000..9dd551c35fb483 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableCacheTest.java @@ -0,0 +1,185 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.iceberg.PartitionSpec; +import org.apache.iceberg.Schema; +import org.apache.iceberg.Table; +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.exceptions.NoSuchTableException; +import org.apache.iceberg.types.Types; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Unit tests for {@link IcebergTableCache} (PERF-01). The cross-query RAW-table cache mirrors + * {@link IcebergLatestSnapshotCache} exactly (same {@link org.apache.doris.connector.cache.MetaCacheEntry} + * backing) but stores the whole {@link Table} instead of the {@code (snapshotId, schemaId)} pin, restoring the + * table-caching half of the legacy {@code IcebergExternalMetaCache}. These tests cover the adapter's contract — + * within-TTL stability, the {@code ttl <= 0} disable, invalidation, and the exception-propagation guarantee the + * partition-view readers depend on. Timed-expiry mechanics are the framework's responsibility (unit-tested in + * the framework module), so they are not re-proven here. + */ +public class IcebergTableCacheTest { + + private static TableIdentifier id() { + return TableIdentifier.of("db", "t"); + } + + /** A distinct fake table, distinguishable by {@link Table#name()}. */ + private static Table table(String name) { + return new FakeIcebergTable(name, + new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get())), + PartitionSpec.unpartitioned(), "s3://b/" + name, Collections.emptyMap()); + } + + @Test + public void cachesWithinTtlAndServesTheSameTable() { + AtomicInteger loads = new AtomicInteger(); + IcebergTableCache c = new IcebergTableCache(100, 1000); + + Table first = c.getOrLoad(id(), () -> { + loads.incrementAndGet(); + return table("first"); + }); + // Second read within TTL must return the CACHED table (first), NOT a freshly-loaded one -> this is what + // lets consecutive queries (and one query's analysis/planning phases) reuse a single load. MUTATION: + // loading live every call -> returns "second" / loads==2 -> red. + Table second = c.getOrLoad(id(), () -> { + loads.incrementAndGet(); + return table("second"); + }); + Assertions.assertEquals("first", first.name()); + Assertions.assertSame(first, second, "within TTL the cached table instance must be served"); + Assertions.assertEquals(1, loads.get(), "the live loader must run exactly once within TTL"); + Assertions.assertTrue(c.isEnabled()); + } + + @Test + public void ttlZeroDisablesCachingAlwaysLive() { + AtomicInteger loads = new AtomicInteger(); + IcebergTableCache c = new IcebergTableCache(0, 1000); + c.getOrLoad(id(), () -> { + loads.incrementAndGet(); + return table("first"); + }); + Table second = c.getOrLoad(id(), () -> { + loads.incrementAndGet(); + return table("second"); + }); + // ttl-second=0 (the no-cache catalog) reads live every time. MUTATION: caching despite ttl<=0 -> + // second=="first" / loads==1 -> red. + Assertions.assertEquals("second", second.name(), "ttl-second=0 must always read the live table"); + Assertions.assertEquals(2, loads.get()); + Assertions.assertFalse(c.isEnabled()); + Assertions.assertEquals(0, c.size(), "ttl-second=0 must not store anything"); + } + + @Test + public void negativeTtlDisablesCachingAlwaysLive() { + // ttl-second=-1 (or any negative) is still the no-cache catalog. Guards the CacheSpec trap where + // ttl == -1 means "no expiration (enabled)": the adapter must translate "<= 0" to disabled. + AtomicInteger loads = new AtomicInteger(); + IcebergTableCache c = new IcebergTableCache(-1, 1000); + c.getOrLoad(id(), () -> { + loads.incrementAndGet(); + return table("first"); + }); + Table second = c.getOrLoad(id(), () -> { + loads.incrementAndGet(); + return table("second"); + }); + Assertions.assertEquals("second", second.name(), "ttl-second=-1 must always read the live table"); + Assertions.assertEquals(2, loads.get()); + Assertions.assertFalse(c.isEnabled()); + } + + @Test + public void invalidateForcesReload() { + AtomicInteger loads = new AtomicInteger(); + IcebergTableCache c = new IcebergTableCache(100, 1000); + c.getOrLoad(id(), () -> { + loads.incrementAndGet(); + return table("first"); + }); + c.invalidate(id()); + // After REFRESH TABLE invalidation the next read goes live. MUTATION: invalidate not clearing -> + // returns cached "first" / loads==1 -> red. + Table after = c.getOrLoad(id(), () -> { + loads.incrementAndGet(); + return table("second"); + }); + Assertions.assertEquals("second", after.name()); + Assertions.assertEquals(2, loads.get()); + } + + @Test + public void invalidateAllClearsEverything() { + IcebergTableCache c = new IcebergTableCache(100, 1000); + c.getOrLoad(TableIdentifier.of("db", "t1"), () -> table("t1")); + c.getOrLoad(TableIdentifier.of("db", "t2"), () -> table("t2")); + Assertions.assertEquals(2, c.size()); + c.invalidateAll(); + Assertions.assertEquals(0, c.size()); + } + + @Test + public void invalidateDbClearsOnlyThatDbsTables() { + AtomicInteger loads = new AtomicInteger(); + IcebergTableCache c = new IcebergTableCache(100, 1000); + c.getOrLoad(TableIdentifier.of("db1", "t1"), () -> table("db1.t1")); + c.getOrLoad(TableIdentifier.of("db1", "t2"), () -> table("db1.t2")); + c.getOrLoad(TableIdentifier.of("db2", "t1"), () -> table("db2.t1")); + Assertions.assertEquals(3, c.size()); + + // REFRESH DATABASE db1 (or a Doris DROP DATABASE db1) must drop BOTH db1 tables and leave db2 intact. + // MUTATION: invalidateDb a no-op -> db1.t1 still cached -> loads stays 0 / after=="db1.t1" -> red. + c.invalidateDb("db1"); + Assertions.assertEquals(1, c.size(), "only db2's single entry must survive"); + + Table afterDb1 = c.getOrLoad(TableIdentifier.of("db1", "t1"), () -> { + loads.incrementAndGet(); + return table("db1.t1.reloaded"); + }); + Assertions.assertEquals("db1.t1.reloaded", afterDb1.name(), "db1.t1 must reload live after invalidateDb"); + Assertions.assertEquals(1, loads.get()); + + Table db2 = c.getOrLoad(TableIdentifier.of("db2", "t1"), () -> { + loads.incrementAndGet(); + return table("db2.t1.reloaded"); + }); + Assertions.assertEquals("db2.t1", db2.name(), "db2 must keep its cached table (not dropped by invalidateDb(db1))"); + Assertions.assertEquals(1, loads.get(), "db2 read must be a hit (no extra load)"); + } + + @Test + public void loaderExceptionPropagatesUnwrapped() { + // The partition-view readers (listPartitions / listPartitionNames) catch NoSuchTableException to degrade + // a concurrent-drop race to an empty list. Routing them through this cache must NOT wrap that exception, + // or the degradation would break and they'd throw instead. The MetaCacheEntry manual-miss-load path + // re-throws the loader's RuntimeException verbatim. MUTATION: wrapping the loader exception -> + // assertThrows(NoSuchTableException) fails (a different type is thrown) -> red. + IcebergTableCache c = new IcebergTableCache(100, 1000); + Assertions.assertThrows(NoSuchTableException.class, () -> c.getOrLoad(id(), () -> { + throw new NoSuchTableException("simulated concurrent drop"); + })); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleTest.java index 2578f13806b925..3080b461a5694b 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleTest.java @@ -18,6 +18,10 @@ package org.apache.doris.connector.iceberg; import com.google.common.collect.ImmutableSet; +import org.apache.iceberg.PartitionSpec; +import org.apache.iceberg.Schema; +import org.apache.iceberg.Table; +import org.apache.iceberg.types.Types; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -25,6 +29,7 @@ import java.io.ByteArrayOutputStream; import java.io.ObjectInputStream; import java.io.ObjectOutputStream; +import java.util.Collections; /** * Tests for {@link IcebergTableHandle}, including the T07 MVCC / time-travel pin carriers and the @@ -354,4 +359,59 @@ public void topnLazyMaterializeComposesWithPinAndScope() { Assertions.assertEquals(42L, h.getSnapshotId()); Assertions.assertEquals(ImmutableSet.of("oss://b/db/t1/f1.parquet"), h.getRewriteFileScope()); } + + // ==================== PERF-01: query-scoped resolved-table memo ==================== + + private static Table fakeTable() { + return new FakeIcebergTable("t1", + new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get())), + PartitionSpec.unpartitioned(), "s3://b/db1/t1", Collections.emptyMap()); + } + + @Test + public void resolvedTableMemoRidesPinAndScopeCopiesButIsNotIdentity() { + Table table = fakeTable(); + IcebergTableHandle bare = new IcebergTableHandle("db1", "t1"); + Assertions.assertNull(bare.getResolvedTable(), "a fresh handle has no resolved-table memo"); + bare.setResolvedTable(table); + Assertions.assertSame(table, bare.getResolvedTable()); + + // WHY (PERF-01): getColumnHandles resolves and memoizes the table BEFORE applySnapshot copies the handle + // to thread the pin on (and rewrite scope / topn likewise). So each with* copy MUST carry the memo, else + // a pin copy would re-load the table it already resolved. MUTATION: a with* copy not carrying + // resolvedTable -> getResolvedTable() null on the copy -> red. + Assertions.assertSame(table, bare.withSnapshot(42L, null, 3L).getResolvedTable()); + Assertions.assertSame(table, bare.withRewriteFileScope(ImmutableSet.of("s3://b/db1/t1/f.parquet")) + .getResolvedTable()); + Assertions.assertSame(table, bare.withTopnLazyMaterialize(true).getResolvedTable()); + + // WHY: the memo is a resolution cache, NOT a coordinate, so it must NOT enter handle identity (the plan + // key). MUTATION: adding resolvedTable to equals/hashCode -> a resolved handle would not equal an + // un-resolved handle for the same table -> the two asserts below fail. + IcebergTableHandle unresolved = new IcebergTableHandle("db1", "t1"); + Assertions.assertEquals(unresolved, bare, "the memo must not change equals"); + Assertions.assertEquals(unresolved.hashCode(), bare.hashCode(), "the memo must not change hashCode"); + } + + @Test + public void resolvedTableMemoIsTransientAcrossSerialization() throws Exception { + IcebergTableHandle original = new IcebergTableHandle("db1", "t1"); + original.setResolvedTable(fakeTable()); + + // The handle serializes even though the memoized FakeIcebergTable is NOT Serializable — proving the + // field is transient. MUTATION: making resolvedTable non-transient -> writeObject throws + // NotSerializableException here -> red. + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (ObjectOutputStream oos = new ObjectOutputStream(baos)) { + oos.writeObject(original); + } + IcebergTableHandle restored; + try (ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(baos.toByteArray()))) { + restored = (IcebergTableHandle) ois.readObject(); + } + // WHY: a deserialized handle (BE side / plan reuse) must start with NO memo and resolve live — the live + // SDK Table is FE-only. MUTATION: a non-transient memo would either fail serialization or staple a stale + // table onto the wire object. + Assertions.assertNull(restored.getResolvedTable(), "the resolved-table memo must not cross serialization"); + } } From 4892c46ceff31e18158c7315df0c64d5dbd6764f Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 00:35:40 +0800 Subject: [PATCH 05/47] [doc](catalog) fe-connector-iceberg PERF-01: summary + tracker updates; next = PERF-02 Record PERF-01 (484f0e0c125) as done: add the implementation summary, mark it complete in tasklist, append the session-2 progress entry (impl + verification + the maven build gotchas), and repoint HANDOFF at PERF-02 (partition-view cross-query cache + MTMV refresh pin), which reuses PERF-01's cache/handle-pin pattern. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 62 +++++++++---------- .../designs/FIX-PERF-01-table-memo-summary.md | 37 +++++++++++ plan-doc/perf-hotpath-iceberg/progress.md | 12 ++++ plan-doc/perf-hotpath-iceberg/tasklist.md | 4 +- 4 files changed, 81 insertions(+), 34 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-01-table-memo-summary.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index 45faaa6afe0ee8..701c2254b49c10 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -6,50 +6,48 @@ --- -# 🆕 下一个 session = **实现 PERF-01（设计已定稿，进入编码）** +# ✅ 上一个 session（session 2，2026-07-18）：**PERF-01 完成并全绿** -## 现状（session 1，2026-07-17） +- **commit `484f0e0c125`**（`[perf]` 一个自包含 commit）。小结见 [`designs/FIX-PERF-01-table-memo-summary.md`](./designs/FIX-PERF-01-table-memo-summary.md)。 +- 落地 = **胖 handle（`IcebergTableHandle.transient resolvedTable`，查询内单实例）+ 跨查询 `IcebergTableCache`（挂 `IcebergConnector`，gate=`isUserSessionEnabled()||restVendedCredentialsEnabled()` 关）**。统一读 helper `IcebergConnectorMetadata.resolveTableForRead`（胖 handle→cache→裸 loadTable）；provider `resolveTable` 胖 handle 优先 + per-call `wrapTableForScan`。Part B 未做（红队证伪）。 +- **验证**：全 iceberg 模块 **932 pass / 0 fail / 1 skip**，checkstyle 绿。度量守门 `planningPassLoadsSameTableOnceViaFatHandle`（规划期同表远端 loadTable = 1，修前 2）。 +- **可复用产物**：`(TableIdentifier)` 键的 `IcebergTableCache` 基建 + 胖 handle 携带模式 + `RecordingIcebergCatalogOps` 计数守门法 —— PERF-02/03/10 直接复用，别再造。 -- PERF-01 **复核 + 红队 + 设计全部完成并定稿**，写入 [`designs/FIX-PERF-01-table-memo-design.md`](./designs/FIX-PERF-01-table-memo-design.md)。**未动任何产品代码**。 -- 设计经用户多轮对齐 Trino 拍板，**最终形态**： - - **① 胖 handle**：`IcebergTableHandle` 加 `transient Table resolvedTable`（不序列化）+ getter/setter；读表统一"胖 handle 优先"；`withSnapshot/withRewriteFileScope/withTopnLazyMaterialize`（`:182/198/207`）**携带前行**；sys 表 handle 不用。→ 查询内单实例、随查询计划自动回收、**连接器侧零每查询累积**。 - - **② 跨查询 `IcebergTableCache`**（新类，仿 `IcebergLatestSnapshotCache`，值=raw Table）：挂长生命周期 `IcebergConnector`，传进每个 fresh metadata + provider，**在读 helper 里消费**（非 catalog-seam 装饰器 → DDL 天然隔离）。**gate=`isUserSessionEnabled() || restVendedCredentialsEnabled()` 关闭**（红队 BLOCKER：vended token 过期会 403）。 - - **Part B（convertPredicate 收窄）已删**（红队证伪为 no-op）。 -- 度量守门：`RecordingIcebergCatalogOps` 断言规划期对同一表 `loadTable` 远端=1；全查询 1（跨查询开）/≤2（关）。 +--- + +# 🆕 下一个 session = **PERF-02（分区视图跨查询缓存 + MTMV refresh pin，C7 C22 C23）** -## 下一个 session 第一件事（TDD，精确到动作） +## 第一件事（立项流程见 README §单项立项流程） -1. 先读 `designs/FIX-PERF-01-table-memo-design.md`（§3 设计 / §4 实现计划 / §5 风险 / §6 度量守门）——这是权威 spec。 -2. **先写测试（TDD）**：`RecordingIcebergCatalogOps`（test 已存在）计数守门——规划期对同一表 `loadTable` 远端次数 = 1（修前基线先记录）。 -3. 按 §4 实现计划 5 步落地（Commit 1，一个 commit）： - - `IcebergTableHandle`：transient `resolvedTable` + getter/setter + 3 个 `with*` 携带。 - - 新增 `IcebergTableCache.java`（仿 `IcebergLatestSnapshotCache`，值 `Table`）。 - - `IcebergConnector`：构造 `tableCache`、传入 metadata/provider、`invalidate*` 三钩子（`:523-553`）、gate。 - - `IcebergConnectorMetadata.loadTable(handle):540` 改"胖 handle 优先→跨查询缓存→remote→setResolvedTable"；`getColumnHandles`/`getTableStatistics`/`resolveTimeTravel`/`getMvccPartitionView`/`listPartitions`/`beginQuerySnapshot` 经它；sys 分支不动。 - - `IcebergScanPlanProvider.resolveTable:1981` 改"胖 handle 优先"+ per-call `wrapTableForScan`；构造加 `tableCache` 参。 -4. 验证（parity + 减负）→ 独立 commit `[perf](catalog) fe-connector-iceberg: (PERF-01)` → 写 `designs/FIX-PERF-01-...-summary.md` → 更新 tasklist/progress/本 HANDOFF。 +1. **复核（动码前，行号信 grep 不信文档）**：按 findings.json C7/C22/C23 重新 grep：分析期 `loadSnapshot → materializeLatest` 对分区表走 `IcebergPartitionUtils.loadRawPartitions`（PARTITIONS 元数据表 `planFiles()+rows()`，读该快照全部 data+delete manifest）；`StatementContext` 只单语句 memoize、跨查询零缓存；MTMV 一次 refresh 里 `isValidRelatedTable/alignMvPartition/generateRelatedPartitionDescs/getAndCopyPartitionItems` 各 materialize 同视图 4~6 次（无 refresh 级 pin + 枚举点间快照偏移风险）。确认成本/乘数仍成立。 +2. **设计**（写 `designs/FIX-PERF-02-...-design.md`）：按 `(TableIdentifier, snapshotId)` 缓存**分区视图**（pin 已在 `beginQuerySnapshot` 后落在 handle 上），挂 `IcebergConnector`（复用 PERF-01 的 `MetaCacheEntry` 基建套路，值=分区视图对象）；MTMV 侧在 `MTMVRefreshContext` 加 refresh 级 `MvccSnapshot` pin。**注意 gate**：分区视图不携带凭证（是元数据），但仍要想清跨查询新鲜度语义是否 = legacy。 +3. **设计红队**（对抗 review，clean-room 偏好）后再实现。 +4. **TDD**：先 `RecordingIcebergCatalogOps`/分区扫描计数守门（分析期同表分区视图 build 从 N→1）。 +5. 参考 PERF-01 的 gate/失效/handle-pin 三板斧，别各造一套 key。 -## ⚠️ 关键认知（别重踩） +## ⚠️ 关键认知（承 PERF-01，别重踩） -- **gate 两半都要**：`isUserSessionEnabled() || restVendedCredentialsEnabled()`（后者独立于前者，红队 Attack 2 = BLOCKER）。关的只是跨查询层；胖 handle 层不 gate（查询内 token 新鲜）。 -- **胖 handle 存 raw Table**；`resolveTable` 命中后仍 per-call `wrapTableForScan`（Kerberos FileIO 不冻进缓存）。 -- **DDL/写/procedure 不碰 resolvedTable、不经读 helper**（走裸 ops 拿 fresh base）。 -- **memo 放连接器/handle 侧，别碰 fe-core 源**（fe-core 只出不进铁律）。`IcebergTableCache` 挂 `IcebergConnector`，`resolvedTable` 挂 `IcebergTableHandle`——都在连接器侧。 -- 携带 transient 的原因：`getColumnHandles` 在 pin 前 set，`pinMvccSnapshot` 用 `withSnapshot` 换 handle，不携带则 pin 后重解析。 +- **胖 handle memo 严格查询内**：`PluginDrivenScanNode.currentHandle` 每查询新建、`resolveConnectorTableHandle`→每次 fresh `getTableHandle`、不挂长生命周期 `ExternalTable`。任何"跨查询"缓存必须挂 `IcebergConnector`（长生命周期），不能靠 handle。 +- **凭证红线**：任何缓存**携带 FileIO 凭证的对象**（raw Table）都要 gate 掉 `session=user`/`vended`；纯元数据（快照 id、分区视图值）不需要，但要确认值里不夹带 FileIO。 +- **测试共享 static handle 会串味**：胖 handle 让 handle 带可变状态，任何跨用例复用的 `static final IcebergTableHandle` 都要改成每用例 fresh（PERF-01 踩过 `T1`）。 --- -# 🧰 构建/验证坑（承自主线 HANDOFF，直接复用） +# 🧰 构建/验证坑（**本轮实证更新，务必照做**） -1. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 被静默跳过，BUILD SUCCESS 是陈旧文件）。 -2. **`mvn ... | tail` 后的 `$?` 是 `tail` 的** —— 读 `BUILD SUCCESS`/`BUILD FAILURE` 行。 -3. **maven 用绝对 `-f /fe/pom.xml`**（cwd 跨调用持久，`cd` 破相对路径）。 -4. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。 -5. **并发 session 共享同一 worktree** —— 动码前探测（`git log/status` + `pgrep maven` 看 `etime`）。 -6. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 +1. **可靠跑单测 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`**（绝对 `-f /fe/pom.xml`）。原因： + - 本 worktree 用 `${revision}` CI 版本，已装的子模块 pom **未 flatten**（parent 仍写 `${revision}`）→ `-pl iceberg` **单模块**永远解析不到 `fe-connector:pom:${revision}`（且 `-Drevision=` 不透传到传递依赖 pom）→ 必须 `-am`（reactor 解析 revision）。 + - `-am` + `test` 相**只到 test**，不产 `fe-connector-hms-hive-shade` 的 shade jar → iceberg 编译报 `HiveConf` 找不到；故用 **`install`**（到 package 相跑 shade）。 + - `-Dtest=` 让**上游模块 0 匹配测试**快速跳过，只 iceberg 跑全套（列表 = `ls src/test/.../*Test.java`）。 +2. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 被静默跳过，BUILD SUCCESS 是陈旧文件）。 +3. **后台 task 通知的 "exit code" 是尾部 `tail`/`echo` 的**，非 maven —— 读日志的 `BUILD SUCCESS`/`Tests run:` 行。 +4. **checkstyle**：LineLength(120) 对 **test 源已 suppress**；主源须 ≤120；import 序 = `SAME_PACKAGE(3)→THIRD_PARTY→STANDARD_JAVA` 组内字母序、组间空行。单独跑 `mvn checkstyle:check -pl ... -am`。 +5. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。 +6. **并发 session 共享同一 worktree** —— 动码前探测（`git log/status` + `pgrep maven` 看 `etime`）。 +7. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 --- # 🗂 开放问题 -- 无。设计定稿，进入实现。 +- 无。PERF-01 已合，进入 PERF-02。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-01-table-memo-summary.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-01-table-memo-summary.md new file mode 100644 index 00000000000000..e67c6cd7acd2fa --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-01-table-memo-summary.md @@ -0,0 +1,37 @@ +# FIX-PERF-01 — Summary（跨查询 Table 缓存 + 查询内胖 handle） + +> 权威设计见 [`FIX-PERF-01-table-memo-design.md`](./FIX-PERF-01-table-memo-design.md)。本文件只记落地结果。 +> 基线 HEAD `b342570d6ca`，分支 `catalog-spi-review-16`。 + +## Problem + +一条带 `WHERE` 的普通 iceberg `SELECT`，规划期把同一张表远程 `loadTable`（metastore RPC + 读 `metadata.json`）**3~7 次**：`getColumnHandles`、`getTableStatistics`、scan provider 的 `resolveTable`（getScanNodeProperties/planScan/streamingSplitEstimate）各自独立 load，分析期 `getMvccPartitionView`/`listPartitions` 再各 load 一次。根因是 SPI 迁移拆散 legacy `IcebergExternalMetaCache` 时只保留了"最新快照 id"那半（`IcebergLatestSnapshotCache`），丢了"缓存 Table 对象"那半。 + +## Fix（一个 commit，iceberg 自包含） + +两层，都在连接器侧（守 fe-core 只出不进铁律）： + +1. **查询内胖 handle**：`IcebergTableHandle` 加 `transient Table resolvedTable` + getter/setter；`withSnapshot/withRewriteFileScope/withTopnLazyMaterialize` 拷贝携带前行。存 **raw** table，不入 equals/hashCode/toString，不序列化。读表统一"胖 handle 优先→解析后回填"。→ 同一查询同一表**规划期恰好 1 次远端 loadTable**（`PluginDrivenScanNode.currentHandle` 每查询新建、随计划回收，连接器侧零累积）。 + +2. **跨查询 `IcebergTableCache`**（新类，仿 `IcebergLatestSnapshotCache`，值=raw Table）：挂长生命周期 `IcebergConnector`，24h TTL、REFRESH 失效，作为胖 handle 首次 miss 的 backing，恢复 legacy 跨查询表缓存。**凭证红线 gate**：`isUserSessionEnabled() || restVendedCredentialsEnabled()` 时置 null 关闭（per-user delegated FileIO 跨用户泄漏；vended token 查询内过期，24h 命中会给 BE 过期 token → 403）。胖 handle 层不受 gate 影响（查询内 token 新鲜）。 + +- 统一读 helper：`IcebergConnectorMetadata.resolveTableForRead`（胖 handle → cache → 裸 `catalogOps.loadTable` → 回填，**不开 auth scope、不包异常**）；`loadTable(handle)` 在 `executeAuthenticated` 内包它；`getMvccPartitionView`/`listPartitions`/`listPartitionNames` 直接在各自 auth scope 内调它，`NoSuchTableException` 原样透传，保住原有"表不存在→空列表"降级。 +- provider `resolveTable`：胖 handle 优先→`IcebergTableCache`(raw)→远端，命中/miss 后都 **per-call `wrapTableForScan`**（Kerberos FileIO 不冻进缓存）。 +- 失效：`IcebergConnector.invalidate{Table,Db,All}` 各加 `tableCache` 失效（与 `latestSnapshotCache` 并列，null-guard）。 +- DDL/写/procedure/sys 表**不经**读 helper（走裸 ops 拿 fresh base）。 +- **Part B（convertPredicate 收窄）未做**（红队证伪为 no-op，见 design §3 Part B）。 + +## Tests（全部新增/加固，`-Dmaven.build.cache.enabled=false`） + +- **度量守门**（`IcebergScanPlanProviderTest.planningPassLoadsSameTableOnceViaFatHandle`）：一条规划把 `getColumnHandles`+`planScan` 穿同一 handle → 远端 `loadTable` 恰好 **1** 次（修前 = 2，胖 handle 前每处各 load）。 +- **胖 handle 携带 + 非身份 + transient**（`IcebergTableHandleTest`）：memo 随三个 `with*` 拷贝前行、不入 equals/hashCode、序列化后为 null（用不可序列化的 `FakeIcebergTable` 证 transient）。 +- **`IcebergTableCacheTest`**（新）：TTL 内命中同实例、`ttl<=0`/负值关缓存、invalidate/invalidateDb/invalidateAll、`NoSuchTableException` 原样透传。 +- **凭证 gate + 失效**（`IcebergConnectorCacheTest`）：plain 目录建缓存且 enabled；vended / session=user 目录 tableCache=null；REFRESH TABLE/DB/CATALOG 逐级清空。 +- **回归修**：`planCount` 原用共享 `static final T1` handle，胖 handle 使其跨用例串味（首个用例的表污染后续）→ 改为每次 fresh handle，镜像生产每查询新 handle。 + +## Result + +- 全 iceberg 模块单测 **932 通过 / 0 失败 / 1 skip**（`install -am`，checkstyle 单独绿）。 +- 行为 parity：现有全部 iceberg 单测不变；胖 handle 是纯去重，快照语义/schema 可见性/时间旅行全对齐 legacy。 +- 减负：每查询规划期远端 `loadTable` 从 3~7 → 1（跨查询开时连续查询命中 → 全查询 1；gate 关时 ≤2）。metastore/REST QPS 除以 3~7。 +- 端到端时延收益（EXPLAIN 实测）留待后续 docker e2e 量化。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index 83d9bbc8bd3a5c..2c46d6aef52eab 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -22,3 +22,15 @@ - **架构澄清**：`IcebergConnectorMetadata`/`IcebergScanPlanProvider`/`IcebergCatalogOps` 均**每调用 new、每查询多个、用完即弃**；仅 `IcebergConnector` + SDK `Catalog` 长生命周期。→ 跨调用共享只能挂 `IcebergConnector` 或 `handle`。 - **用户定稿设计（多轮对齐 Trino）**：**胖 handle（`IcebergTableHandle.transient resolvedTable`，查询内单实例、随查询计划自动回收、连接器侧零累积，= Trino per-transaction 胖 handle 的 Doris 对应物）+ 跨查询 `IcebergTableCache`（挂 `IcebergConnector`，仿 `IcebergLatestSnapshotCache`，读 helper 消费非 catalog-seam 装饰器→DDL 天然隔离，gate=`isUserSessionEnabled()||restVendedCredentialsEnabled()`）**。去重计数：规划期 1 次、全查询 1 次（跨查询开）/≤2（关）。全设计 + 度量守门见 `designs/FIX-PERF-01-table-memo-design.md`。 - **未动任何产品代码**。**下一步**：实现（TDD，先 `RecordingIcebergCatalogOps` 度量守门）。 + +--- + +## 2026-07-18 — session 2：PERF-01 实现 + 全绿（commit `484f0e0c125`） + +- **实现按定稿落地（TDD，一个 `[perf]` commit）**：① `IcebergTableHandle` 加 `transient Table resolvedTable`（不序列化、不入 equals/hashCode，三个 `with*` 携带前行）；② 新增 `IcebergTableCache`（仿 `IcebergLatestSnapshotCache`，值 raw Table，manual-miss-load 原样透传异常）；③ `IcebergConnector` 按 gate 建 tableCache 或 null，传入 metadata/provider，三个 `invalidate*` 加跨查询失效；④ metadata 统一读 helper `resolveTableForRead`（胖 handle→cache→裸 loadTable，不开 auth scope/不包异常），`loadTable(handle)` 包 auth，`getMvccPartitionView`/`listPartitions`/`listPartitionNames` 经它并保住 `NoSuchTableException`→空降级；⑤ provider `resolveTable` 胖 handle 优先 + per-call `wrapTableForScan`。 +- **关键校验（动码前）**：确认 `PluginDrivenScanNode.currentHandle` 每查询新建、`resolveConnectorTableHandle`→每次 fresh `getTableHandle`、不挂长生命周期 `ExternalTable` → 胖 handle memo 严格查询内、无跨查询泄漏（design 假设成立，非一厢情愿）。`MetaCacheEntry.loadAndTrack` 确认 `throw e`（RuntimeException 原样），故 `NoSuchTableException` 穿缓存不被包 → 降级不破。 +- **度量守门（新）**：`IcebergScanPlanProviderTest.planningPassLoadsSameTableOnceViaFatHandle` —— `getColumnHandles`+`planScan` 穿同一 handle → 远端 loadTable 恰 1 次（修前 2）。另加 handle 携带/transient、`IcebergTableCacheTest`、凭证 gate + REFRESH 失效诸测。 +- **回归修（1 个）**：`IcebergScanPlanProviderTest` 的 COUNT(*) 用例共享 `static final T1` handle，胖 handle 令其跨用例串味（空表用例污染后续 → 5 挂）→ 改 `planCount` 每次 fresh handle（镜像生产每查询新 handle）。**非生产 bug**，纯测试 artifact。 +- **结果**：全 iceberg 模块 **932 pass / 0 fail / 1 skip**（`install -am`），checkstyle 绿。summary 见 `designs/FIX-PERF-01-table-memo-summary.md`。 +- **构建坑记录**：本 worktree `${revision}` CI 版本 + 未 flatten 的已装 pom → `-pl iceberg` 单模块永远解析不到 `fe-connector:pom:${revision}`（`-Drevision=` 不透传到传递依赖 pom）；且 `-am test` 只到 test 相不产 hms-hive-shade 的 shade jar（缺 `HiveConf`）。**可靠跑法 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest= -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`**（reactor 解析 revision + 到 install 相产 shade jar + 上游 0 匹配测试快速跳过）。 +- **下一步**：见 HANDOFF —— PERF-02（分区视图跨查询缓存 + MTMV refresh pin，复用本任务的 `(table, snapshotId)` pin 与胖 handle 模式）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index d67fdcb6dfda09..b60e971b84974b 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -15,7 +15,7 @@ | ID | 优先级 | 覆盖发现 | 主题（一句话） | 依赖 | 状态 | commit | |---|---|---|---|---|---|---| -| PERF-01 | P0 | C1 C4 C6 C10 C16 | 一次规划 3~7 次远程 loadTable → 胖 handle(查询内单实例)+ 跨查询 IcebergTableCache(挂 Connector);~~convertPredicate 收窄~~已删(红队证伪) | — | 🚧 设计定稿,待实现 | | +| PERF-01 | P0 | C1 C4 C6 C10 C16 | 一次规划 3~7 次远程 loadTable → 胖 handle(查询内单实例)+ 跨查询 IcebergTableCache(挂 Connector);~~convertPredicate 收窄~~已删(红队证伪) | — | ✅ 完成 | `484f0e0c125` | | PERF-02 | P0 | C7 C22 C23 | 分区视图每查询重扫 PARTITIONS 元数据表 → `(table,snapshotId)` 缓存 + MTMV refresh pin | 与 01 共享快照 pin 机制 | ⏳ | | | PERF-03 | P0 | C2 C11 | #64134 复活：`file_format_type` 兜底走整表 planFiles → memoize / 从枚举反推 | 受益于 01 的失效收窄（消第二次） | ⏳ | | | PERF-04 | P1 | C17 C18 | streaming / COUNT(*) 下推旁路 IcebergManifestCache → 两条旁路接回 | — | ⏳ | | @@ -31,7 +31,7 @@ ## P0 — 无缓存/兜底类，触发面=所有 iceberg 查询 -### [ ] PERF-01 — 簇1：per-planning-pass Table memo（C1 C4 C6 C10 C16） +### [x] PERF-01 — 簇1：胖 handle(查询内) + 跨查询 IcebergTableCache（C1 C4 C6 C10 C16） · ✅ `484f0e0c125` - **病灶**：`IcebergCatalogOps.loadTable:340` 是对 SDK `catalog.loadTable()` 的裸委派（每次 = metastore RPC + metadata.json 读）；全仓无 `CachingCatalog`，`IcebergLatestSnapshotCache` 只存 `(snapshotId,schemaId)` 不存 Table。SPI 各入口各自 load ⇒ 一次带 WHERE 的查询 **3~7 次** loadTable（7 个调用点见报告簇1 表）。放大器 = `convertPredicate:795-798` **无条件**清空 `cachedPropertiesResult` ⇒ 整份 properties（loadTable/format/schema thrift+base64/凭证 overlay，全是 filter 不变量）第二次重算。 - **修复方向**：以 `beginQuerySnapshot` 已有的 pin 为天然 key，做 per-planning-pass 的 **Table memo**（键 `(TableIdentifier, pinnedSnapshotId)`，挂长生命周期 `IcebergConnector` 或随 handle 传递），让 `getColumnHandles / resolveTable / streamingSplitEstimate / planScan` 共享一次 load；同时把 `convertPredicate` 失效**收窄**到真正依赖 conjunct 的 prop（谓词对 properties 的唯一影响仅 pushdown-predicates 一个 prop）。 - **收益**：每查询 −3~6 次远程往返 ≈ 规划延迟 −0.2~1.5s；metastore/REST QPS 除以 3~7。 From 6ab2f8bea81b2a8b90390bb9adfa8ee608c7cdbb Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 01:46:12 +0800 Subject: [PATCH 06/47] [perf](catalog) fe-connector-iceberg: cache the partition-view scan across queries (PERF-02) For a partitioned table, every query pays an analysis-phase PARTITIONS metadata-table scan (IcebergPartitionUtils.loadRawPartitions -> partitionsTable.newScan().useSnapshot(id).planFiles()+rows()), which the iceberg SDK materializes by reading EVERY data+delete manifest of the snapshot (O(#manifests) remote IO, independent of query selectivity). It ran once per query (memoized only within the statement) and 4~6 times per MTMV refresh, with no cross-query cache -- the partition-info half of the legacy IcebergExternalMetaCache that the SPI cutover dropped. PERF-01 already deduped the loadTable in this cluster; this fixes the remaining scan. Add a connector-side IcebergPartitionCache keyed by (TableIdentifier, snapshotId), value = the raw partition list. The three consumers -- buildMvccPartitionView (MVCC/MTMV view), listPartitions (selectedPartitionNum), listPartitionNames (SHOW PARTITIONS) -- all funnel through loadRawPartitions, so they share one scan per (table, snapshot). The snapshot id resolved inside those methods is held stable across queries (and across one refresh's enumeration points) by IcebergLatestSnapshotCache, so the key is stable and the MTMV 4~6x re-enumeration collapses onto one scan too. A snapshot is immutable, so the cached value is a pure function of the key (always correct; a new commit -> new snapshot id -> new key -> a live scan). The value is pure metadata with NO FileIO/credential, so unlike the PERF-01 table cache it needs no credential gate and is built for every catalog (only the meta.cache.iceberg.table.ttl-second knob disables it). The cached list is unmodifiable; the loader's exception propagates verbatim so listPartitions keeps its dropped-partition-source-column degradation, and a failed scan is not cached. REFRESH TABLE/DB/CATALOG invalidate it alongside the other caches. The MTMV refresh-level MvccSnapshot pin the audit also suggested is intentionally NOT done: IcebergLatestSnapshotCache already stabilizes the snapshot across a refresh (ttl>0), so this cache collapses the repeats without a fe-core change (the only residual is the ttl=0 no-cache catalog's pre-existing enumeration skew, which is out of scope for a connector-side perf fix). Metric gate: a real partitioned table + a real cache; repeated buildMvccPartitionView plus listPartitions at one snapshot run the PARTITIONS scan exactly once (cache.loadCountForTest()==1, was per-call). Full iceberg module suite: 943 pass / 0 fail / 1 skip; checkstyle clean. Behavior parity: cached enumeration equals a live one; snapshot/schema visibility unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../connector/iceberg/IcebergConnector.java | 17 +- .../iceberg/IcebergConnectorMetadata.java | 24 ++- .../iceberg/IcebergPartitionCache.java | 142 +++++++++++++ .../iceberg/IcebergPartitionUtils.java | 52 ++++- .../iceberg/IcebergConnectorCacheTest.java | 54 +++++ .../iceberg/IcebergPartitionCacheTest.java | 186 ++++++++++++++++++ .../iceberg/IcebergPartitionUtilsTest.java | 29 +++ 7 files changed, 493 insertions(+), 11 deletions(-) create mode 100644 fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionCache.java create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergPartitionCacheTest.java diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java index ff6567d124eab7..6a86035c6795f2 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java @@ -164,6 +164,10 @@ public class IcebergConnector implements Connector { // (iceberg.rest.session=user / REST vended-credentials) — see the constructor. The query-scoped fat handle // (IcebergTableHandle.resolvedTable) is always on and independent of this field. private final IcebergTableCache tableCache; + // PERF-02: cross-query partition-view cache (the raw PARTITIONS-scan result, keyed by (table, snapshotId)). + // Built unconditionally — the cached value is pure metadata with no FileIO/credential, so no credential gate + // (unlike tableCache); only the TTL knob disables it. + private final IcebergPartitionCache partitionCache; private final IcebergManifestCache manifestCache = new IcebergManifestCache(); // commit-bridge supply (S4 part 2): per-catalog stash carrying a row-level DML's non-equality delete supply // across the scan->write seam — the scan provider fills it (keyed by queryId), the write provider drains it @@ -205,6 +209,9 @@ public IcebergConnector(Map properties, ConnectorContext context ? null : new IcebergTableCache( resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY); + // PERF-02: partition-view cache. Pure metadata (no credentials) -> no gate; same TTL/capacity as above. + this.partitionCache = new IcebergPartitionCache( + resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY); } /** @@ -229,7 +236,7 @@ static long resolveTableCacheTtlSecond(Map properties) { @Override public ConnectorMetadata getMetadata(ConnectorSession session) { return new IcebergConnectorMetadata( - newCatalogBackedOps(session), properties, context, latestSnapshotCache, tableCache); + newCatalogBackedOps(session), properties, context, latestSnapshotCache, tableCache, partitionCache); } /** @@ -543,6 +550,7 @@ public void invalidateTable(String dbName, String tableName) { if (tableCache != null) { tableCache.invalidate(TableIdentifier.of(dbName, tableName)); } + partitionCache.invalidate(TableIdentifier.of(dbName, tableName)); } /** @@ -562,6 +570,7 @@ public void invalidateDb(String dbName) { if (tableCache != null) { tableCache.invalidateDb(dbName); } + partitionCache.invalidateDb(dbName); } /** @@ -577,6 +586,7 @@ public void invalidateAll() { if (tableCache != null) { tableCache.invalidateAll(); } + partitionCache.invalidateAll(); manifestCache.invalidateAll(); } @@ -611,6 +621,11 @@ IcebergTableCache tableCacheForTest() { return tableCache; } + /** Test-only: the cross-query partition-view cache (PERF-02; always built, no credential gate). */ + IcebergPartitionCache partitionCacheForTest() { + return partitionCache; + } + @Override public ConnectorScanPlanProvider getScanPlanProvider() { // Mirrors PaimonConnector.getScanPlanProvider: build a fresh provider per call over the lazily-built diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index 510d552fef172a..f24ad5541d9b08 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -148,26 +148,37 @@ public class IcebergConnectorMetadata implements ConnectorMetadata { // and the credential-gated catalogs pass null; the query-scoped fat handle (IcebergTableHandle) works // regardless. Consumed only by resolveTableForRead. private final IcebergTableCache tableCache; + // PERF-02: cross-query partition-view cache (null = no cross-query layer; the convenience ctors used by + // direct-construction tests pass null). Consumed by getMvccPartitionView / listPartitions / listPartitionNames. + private final IcebergPartitionCache partitionCache; public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties, ConnectorContext context) { - this(catalogOps, properties, context, new IcebergLatestSnapshotCache(0L, 1), null); + this(catalogOps, properties, context, new IcebergLatestSnapshotCache(0L, 1), null, null); } /** Convenience ctor without a cross-query table cache (tableCache null); used by MVCC/statistics tests. */ public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties, ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache) { - this(catalogOps, properties, context, latestSnapshotCache, null); + this(catalogOps, properties, context, latestSnapshotCache, null, null); } + /** Convenience ctor without a partition-view cache (partitionCache null). */ public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties, ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache, IcebergTableCache tableCache) { + this(catalogOps, properties, context, latestSnapshotCache, tableCache, null); + } + + public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties, + ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache, + IcebergTableCache tableCache, IcebergPartitionCache partitionCache) { this.catalogOps = catalogOps; this.properties = properties; this.context = context; this.latestSnapshotCache = latestSnapshotCache; this.tableCache = tableCache; + this.partitionCache = partitionCache; } // ========== ConnectorSchemaOps ========== @@ -1490,7 +1501,8 @@ public Optional getMvccPartitionView( try { return context.executeAuthenticated(() -> { Table table = resolveTableForRead(iceHandle); - return Optional.of(IcebergPartitionUtils.buildMvccPartitionView(table, iceHandle.getSnapshotId())); + return Optional.of(IcebergPartitionUtils.buildMvccPartitionView(table, iceHandle.getSnapshotId(), + TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache)); }); } catch (Exception e) { throw new RuntimeException("Failed to build iceberg MVCC partition view, error message is:" @@ -1518,7 +1530,8 @@ public List listPartitionNames(ConnectorSession session, ConnectorTableH iceHandle.getDbName(), iceHandle.getTableName(), e); return Collections.emptyList(); } - return IcebergPartitionUtils.listPartitionNames(table); + return IcebergPartitionUtils.listPartitionNames(table, + TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache); }); } catch (Exception e) { throw new RuntimeException("Failed to list iceberg partition names, error message is:" @@ -1549,7 +1562,8 @@ public List listPartitions(ConnectorSession session, iceHandle.getDbName(), iceHandle.getTableName(), e); return Collections.emptyList(); } - return IcebergPartitionUtils.listPartitions(table); + return IcebergPartitionUtils.listPartitions(table, + TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache); }); } catch (Exception e) { throw new RuntimeException("Failed to list iceberg partitions, error message is:" diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionCache.java new file mode 100644 index 00000000000000..3d68ab664cfb87 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionCache.java @@ -0,0 +1,142 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.cache.CacheSpec; +import org.apache.doris.connector.cache.MetaCacheEntry; +import org.apache.doris.connector.iceberg.IcebergPartitionUtils.IcebergRawPartition; + +import org.apache.iceberg.catalog.Namespace; +import org.apache.iceberg.catalog.TableIdentifier; + +import java.util.List; +import java.util.Objects; +import java.util.concurrent.ForkJoinPool; +import java.util.function.Supplier; + +/** + * Per-catalog cache of an iceberg table's raw partition list (PERF-02), keyed by {@code (TableIdentifier, + * snapshotId)}. Restores the partition-info half of the legacy {@code IcebergExternalMetaCache} that the SPI + * cutover dropped: the analysis-phase PARTITIONS metadata-table scan + * ({@link IcebergPartitionUtils#loadRawPartitionsUncached}, which the iceberg SDK materializes by reading EVERY + * data+delete manifest of the snapshot) was re-run per query and re-run 4~6 times per MTMV refresh, with no + * cross-query reuse. The three consumers ({@code buildMvccPartitionView} for the MVCC/MTMV partition view, + * {@code listPartitions} for {@code selectedPartitionNum}, {@code listPartitionNames} for SHOW PARTITIONS) all + * funnel through it, so they share a single scan per {@code (table, snapshot)}. + * + *

Snapshot-keyed, so always correct. A snapshot is immutable, so the derived partitions are a pure + * function of the key; a new commit yields a new snapshot id (a new key -> a live scan). Within the TTL the + * snapshot id itself is held stable by {@link IcebergLatestSnapshotCache}, which is what makes the key stable + * across queries and across the enumeration points of one MTMV refresh. + * + *

No credential gate (unlike {@link IcebergTableCache}): the cached value is pure metadata (partition + * names, values, transforms, timestamps, snapshot ids) and carries no {@code FileIO} / credential, so it is + * safe to share across users and is built unconditionally (only the TTL knob disables it). + * + *

Backed identically to {@link IcebergLatestSnapshotCache}: a contextual, access-TTL {@link MetaCacheEntry} + * with manual miss-load, so the scan runs OUTSIDE Caffeine's compute lock and its exception (e.g. the + * dropped-partition-source-column {@link org.apache.iceberg.exceptions.ValidationException} that + * {@code listPartitions} degrades on) propagates verbatim and a failed scan is not cached. TTL is + * {@code meta.cache.iceberg.table.ttl-second}; {@code <= 0} disables (read live). Lives on the long-lived + * per-catalog {@link IcebergConnector}; a REFRESH CATALOG rebuilds the connector and thus the cache. + */ +final class IcebergPartitionCache { + + /** Immutable composite key: a table's partition list is distinct per pinned snapshot id. */ + static final class Key { + final TableIdentifier id; + final long snapshotId; + + Key(TableIdentifier id, long snapshotId) { + this.id = id; + this.snapshotId = snapshotId; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof Key)) { + return false; + } + Key that = (Key) o; + return snapshotId == that.snapshotId && Objects.equals(id, that.id); + } + + @Override + public int hashCode() { + return Objects.hash(id, snapshotId); + } + } + + private final MetaCacheEntry> entry; + + IcebergPartitionCache(long ttlSeconds, int maxSize) { + // Mirror IcebergLatestSnapshotCache: translate the connector's "<= 0 disables" contract to CacheSpec's + // ttl == 0 (disabled) rather than passing a negative value through (which CacheSpec reads as "no + // expiration"). + CacheSpec spec = ttlSeconds > 0 + ? CacheSpec.of(true, ttlSeconds, maxSize) + : CacheSpec.of(true, CacheSpec.CACHE_TTL_DISABLE_CACHE, maxSize); + this.entry = new MetaCacheEntry<>("iceberg-partition", null, spec, + ForkJoinPool.commonPool(), false, true, 0L, true); + } + + /** Caching is on only when the TTL is positive; ttl-second <= 0 means "always scan live". */ + boolean isEnabled() { + return entry.stats().isEffectiveEnabled(); + } + + /** + * Returns the cached raw partition list for {@code key} if present and unexpired, else runs {@code loader} + * (the live PARTITIONS scan), caches and returns it. Disabled cache -> {@code loader} every call. The + * loader runs OUTSIDE Caffeine's compute lock (single-flight per key) and its exception propagates unwrapped. + */ + List getOrLoad(Key key, Supplier> loader) { + return entry.get(key, ignored -> loader.get()); + } + + /** Drops every cached snapshot entry for one table so the next read scans live (REFRESH TABLE). */ + void invalidate(TableIdentifier id) { + entry.invalidateIf(key -> key.id.equals(id)); + } + + /** Drops every cached entry for one database (REFRESH DATABASE / DROP DATABASE); db match = namespace equality. */ + void invalidateDb(String dbName) { + Namespace ns = Namespace.of(dbName); + entry.invalidateIf(key -> key.id.namespace().equals(ns)); + } + + /** Drops all cached entries (REFRESH CATALOG). */ + void invalidateAll() { + entry.invalidateAll(); + } + + /** Test-only: current number of cached entries (accurate map membership, not Caffeine's estimate). */ + int size() { + int[] count = {0}; + entry.forEach((key, value) -> count[0]++); + return count[0]; + } + + /** Test-only: how many times the live loader (the PARTITIONS scan) actually ran — the metric gate. */ + long loadCountForTest() { + return entry.stats().getLoadSuccessCount(); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java index 45b191180670c1..31767a03c4aef2 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java @@ -36,6 +36,7 @@ import org.apache.iceberg.Snapshot; import org.apache.iceberg.StructLike; import org.apache.iceberg.Table; +import org.apache.iceberg.catalog.TableIdentifier; import org.apache.iceberg.exceptions.ValidationException; import org.apache.iceberg.io.CloseableIterable; import org.apache.iceberg.types.Type; @@ -530,6 +531,16 @@ static List parsePartitionValuesFromJson(String partitionDataJson) { * handle), or {@code < 0} to enumerate at the table's current (latest) snapshot. */ static ConnectorMvccPartitionView buildMvccPartitionView(Table table, long pinnedSnapshotId) { + return buildMvccPartitionView(table, pinnedSnapshotId, null, null); + } + + /** + * Cache-aware overload (PERF-02): {@code id} + {@code cache} route the PARTITIONS scan through the + * per-catalog {@link IcebergPartitionCache} keyed by {@code (id, resolvedSnapshotId)}. {@code cache == null} + * reads live (offline tests / no-cache catalog). + */ + static ConnectorMvccPartitionView buildMvccPartitionView(Table table, long pinnedSnapshotId, + TableIdentifier id, IcebergPartitionCache cache) { if (!isValidRelatedTable(table)) { return ConnectorMvccPartitionView.unpartitioned(); } @@ -562,7 +573,7 @@ static ConnectorMvccPartitionView buildMvccPartitionView(Table table, long pinne // two rows rendering the same field=value name have byte-identical ranges, and feeding both to the merge // would make the name self-enclose and be dropped (0 partitions where master keeps 1). Map allByName = new LinkedHashMap<>(); - for (IcebergRawPartition raw : loadRawPartitions(table, snapshotId)) { + for (IcebergRawPartition raw : loadRawPartitions(id, table, snapshotId, cache)) { RangeBuild rb = buildRange(raw.name, raw.values.get(0), raw.transforms.get(0), sourceType, raw.lastUpdateTime, raw.lastSnapshotId); allByName.put(rb.name, rb); @@ -602,6 +613,12 @@ static ConnectorMvccPartitionView buildMvccPartitionView(Table table, long pinne * unpartitioned or empty table yields an empty list. Must run inside {@code context.executeAuthenticated}. */ static List listPartitionNames(Table table) { + return listPartitionNames(table, null, null); + } + + /** Cache-aware overload (PERF-02): see {@link #buildMvccPartitionView(Table, long, TableIdentifier, + * IcebergPartitionCache)}. Keyed by the table's CURRENT snapshot id. */ + static List listPartitionNames(Table table, TableIdentifier id, IcebergPartitionCache cache) { if (table.spec().isUnpartitioned()) { return Collections.emptyList(); } @@ -609,7 +626,7 @@ static List listPartitionNames(Table table) { if (current == null) { return Collections.emptyList(); } - List raws = loadRawPartitions(table, current.snapshotId()); + List raws = loadRawPartitions(id, table, current.snapshotId(), cache); List names = new ArrayList<>(raws.size()); for (IcebergRawPartition raw : raws) { names.add(raw.name); @@ -629,6 +646,12 @@ static List listPartitionNames(Table table) { * {@code context.executeAuthenticated}. */ static List listPartitions(Table table) { + return listPartitions(table, null, null); + } + + /** Cache-aware overload (PERF-02): see {@link #buildMvccPartitionView(Table, long, TableIdentifier, + * IcebergPartitionCache)}. Keyed by the table's CURRENT snapshot id. */ + static List listPartitions(Table table, TableIdentifier id, IcebergPartitionCache cache) { if (table.spec().isUnpartitioned()) { return Collections.emptyList(); } @@ -638,7 +661,7 @@ static List listPartitions(Table table) { } List raws; try { - raws = loadRawPartitions(table, current.snapshotId()); + raws = loadRawPartitions(id, table, current.snapshotId(), cache); } catch (ValidationException e) { if (!isDroppedPartitionSourceColumn(e)) { // A different iceberg validation error (not the dropped-partition-source-column case) — fail loud. @@ -706,7 +729,26 @@ static boolean isValidRelatedTable(Table table) { * {@code last_updated_at} (row 9) / {@code last_updated_snapshot_id} (row 10) are optional, so a missing * value (NPE on the typed getter) degrades to {@code 0} / {@code UNKNOWN_SNAPSHOT_ID}, exactly like master. */ - private static List loadRawPartitions(Table table, long snapshotId) { + /** + * The cross-query PARTITIONS-scan de-duplication seam (PERF-02): when {@code cache} is non-null the raw + * partition list is served from / populated into the per-catalog {@link IcebergPartitionCache} keyed by + * {@code (id, snapshotId)} — a snapshot is immutable, so the derived partitions are a pure function of that + * key and safe to reuse across queries (restoring the legacy IcebergExternalMetaCache partition-info cache). + * A {@code null} cache (offline unit tests / the no-cache catalog) reads live every call. The cached list is + * unmodifiable so a shared entry cannot be mutated by a concurrent reader; the loader's exception (e.g. the + * dropped-partition-source-column {@link ValidationException}) propagates verbatim so callers keep their own + * degradation, and a failed scan is not cached. + */ + private static List loadRawPartitions(TableIdentifier id, Table table, long snapshotId, + IcebergPartitionCache cache) { + if (cache == null) { + return loadRawPartitionsUncached(table, snapshotId); + } + return cache.getOrLoad(new IcebergPartitionCache.Key(id, snapshotId), + () -> Collections.unmodifiableList(loadRawPartitionsUncached(table, snapshotId))); + } + + private static List loadRawPartitionsUncached(Table table, long snapshotId) { Table partitionsTable = MetadataTableUtils.createMetadataTableInstance(table, MetadataTableType.PARTITIONS); List partitions = new ArrayList<>(); try (CloseableIterable tasks = partitionsTable.newScan().useSnapshot(snapshotId).planFiles()) { @@ -902,7 +944,7 @@ static long latestSnapshotId(String name, Map> mergeMap, } /** One PARTITIONS-metadata-table row reduced to what the MTMV partition view needs (port of IcebergPartition). */ - private static final class IcebergRawPartition { + static final class IcebergRawPartition { private final String name; // Partition-field SOURCE column names (lowercased), parallel to {@link #values}, so listPartitions can // build a value map keyed by the generic partition-column remote name (see IcebergConnectorMetadata diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java index 0f5c12742f4f47..2b833db7c92149 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java @@ -238,4 +238,58 @@ public void refreshHooksInvalidateCrossQueryTableCache() { connector.invalidateAll(); Assertions.assertEquals(0, cache.size(), "REFRESH CATALOG drops everything"); } + + // ==================== PERF-02: partition-view cache (no gate) + invalidation ==================== + + private static IcebergPartitionCache.Key partKey(String db, String tbl, long snapshotId) { + return new IcebergPartitionCache.Key(TableIdentifier.of(db, tbl), snapshotId); + } + + @Test + public void partitionCacheBuiltForAllCatalogsIncludingCredentialGatedOnes() { + // The partition-view cache stores pure metadata (no FileIO/credential), so unlike the table cache it is + // built for EVERY catalog -- including per-user session and REST vended-credentials, which disable the + // table cache. MUTATION: gating the partition cache on the credential flags -> null here -> red. + Assertions.assertNotNull( + new IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext()).partitionCacheForTest(), + "a plain catalog builds the partition cache"); + Map vended = new HashMap<>(); + vended.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST); + vended.put(IcebergConnectorProperties.REST_VENDED_CREDENTIALS_ENABLED, "true"); + Assertions.assertNotNull( + new IcebergConnector(vended, new RecordingConnectorContext()).partitionCacheForTest(), + "a vended-credentials catalog still builds the partition cache (metadata carries no credentials)"); + Map session = new HashMap<>(); + session.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST); + session.put(IcebergConnectorProperties.REST_SESSION, IcebergConnectorProperties.SESSION_USER); + Assertions.assertNotNull( + new IcebergConnector(session, new RecordingConnectorContext()).partitionCacheForTest(), + "a per-user session catalog still builds the partition cache"); + } + + @Test + public void refreshHooksInvalidatePartitionCache() { + // The REFRESH hooks must clear the partition-view cache too (else external DDL/writes would stay invisible + // beyond the pin): REFRESH TABLE drops that table's snapshot entries, REFRESH DATABASE that db's, REFRESH + // CATALOG everything. MUTATION: an invalidate* hook not touching partitionCache -> a stale entry survives + // -> a size assert below red. + IcebergConnector connector = + new IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext()); + IcebergPartitionCache cache = connector.partitionCacheForTest(); + Assertions.assertNotNull(cache); + cache.getOrLoad(partKey("db1", "t1", 1L), Collections::emptyList); + cache.getOrLoad(partKey("db1", "t1", 2L), Collections::emptyList); + cache.getOrLoad(partKey("db1", "t2", 1L), Collections::emptyList); + cache.getOrLoad(partKey("db2", "t1", 1L), Collections::emptyList); + Assertions.assertEquals(4, cache.size()); + + connector.invalidateTable("db1", "t1"); + Assertions.assertEquals(2, cache.size(), "REFRESH TABLE drops both snapshot entries of db1.t1"); + + connector.invalidateDb("db1"); + Assertions.assertEquals(1, cache.size(), "REFRESH DATABASE drops db1's remaining table"); + + connector.invalidateAll(); + Assertions.assertEquals(0, cache.size(), "REFRESH CATALOG drops everything"); + } } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergPartitionCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergPartitionCacheTest.java new file mode 100644 index 00000000000000..576d46151c7a42 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergPartitionCacheTest.java @@ -0,0 +1,186 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.iceberg.IcebergPartitionUtils.IcebergRawPartition; + +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.exceptions.ValidationException; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Unit tests for {@link IcebergPartitionCache} (PERF-02). Mirrors {@link IcebergTableCacheTest} but keys by + * {@code (TableIdentifier, snapshotId)} and stores the raw partition list. Covers within-TTL stability, the + * {@code ttl <= 0} disable, invalidation, and the exception-propagation guarantee that {@code listPartitions}' + * dropped-partition-source-column degradation depends on. + */ +public class IcebergPartitionCacheTest { + + private static IcebergPartitionCache.Key key(String db, String tbl, long snapshotId) { + return new IcebergPartitionCache.Key(TableIdentifier.of(db, tbl), snapshotId); + } + + /** A raw partition list of the given size, distinguishable by size. */ + private static List raws(int n) { + List list = new java.util.ArrayList<>(); + for (int i = 0; i < n; i++) { + list.add(new IcebergRawPartition("p" + i, Collections.singletonList("c"), + Collections.singletonList("v" + i), Collections.singletonList("identity"), 0L, 0L)); + } + return list; + } + + @Test + public void cachesWithinTtlAndServesTheSameList() { + AtomicInteger loads = new AtomicInteger(); + IcebergPartitionCache c = new IcebergPartitionCache(100, 1000); + + List first = c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return raws(3); + }); + // Second read of the SAME (table, snapshot) within TTL returns the cached list, NOT a fresh scan -> this + // is what collapses the per-query and per-MTMV-refresh PARTITIONS scans. MUTATION: scanning live every + // call -> returns the 7-element list / loads==2 -> red. + List second = c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return raws(7); + }); + Assertions.assertEquals(3, first.size()); + Assertions.assertSame(first, second, "within TTL the cached partition list must be served"); + Assertions.assertEquals(1, loads.get(), "the live scan must run exactly once within TTL"); + Assertions.assertEquals(1, c.loadCountForTest(), "the metric-gate load count must be 1"); + Assertions.assertTrue(c.isEnabled()); + } + + @Test + public void differentSnapshotIdIsADifferentKey() { + AtomicInteger loads = new AtomicInteger(); + IcebergPartitionCache c = new IcebergPartitionCache(100, 1000); + c.getOrLoad(key("db", "t", 1L), () -> { + loads.incrementAndGet(); + return raws(1); + }); + // A new commit yields a new snapshot id -> a distinct key -> a live scan (freshness across snapshots). + // MUTATION: keying by table only (ignoring snapshotId) -> loads==1 / serves the snapshot-1 list -> red. + List s2 = c.getOrLoad(key("db", "t", 2L), () -> { + loads.incrementAndGet(); + return raws(9); + }); + Assertions.assertEquals(9, s2.size()); + Assertions.assertEquals(2, loads.get()); + Assertions.assertEquals(2, c.size()); + } + + @Test + public void ttlZeroDisablesCachingAlwaysLive() { + AtomicInteger loads = new AtomicInteger(); + IcebergPartitionCache c = new IcebergPartitionCache(0, 1000); + c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return raws(3); + }); + List second = c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return raws(7); + }); + // ttl-second=0 (the no-cache catalog) scans live every time. MUTATION: caching despite ttl<=0 -> + // second.size()==3 / loads==1 -> red. + Assertions.assertEquals(7, second.size(), "ttl-second=0 must always scan live"); + Assertions.assertEquals(2, loads.get()); + Assertions.assertFalse(c.isEnabled()); + Assertions.assertEquals(0, c.size(), "ttl-second=0 must not store anything"); + } + + @Test + public void negativeTtlDisablesCachingAlwaysLive() { + AtomicInteger loads = new AtomicInteger(); + IcebergPartitionCache c = new IcebergPartitionCache(-1, 1000); + c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return raws(3); + }); + List second = c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return raws(7); + }); + Assertions.assertEquals(7, second.size(), "ttl-second=-1 must always scan live"); + Assertions.assertEquals(2, loads.get()); + Assertions.assertFalse(c.isEnabled()); + } + + @Test + public void invalidateDropsAllSnapshotsOfOneTable() { + AtomicInteger loads = new AtomicInteger(); + IcebergPartitionCache c = new IcebergPartitionCache(100, 1000); + c.getOrLoad(key("db", "t", 1L), () -> raws(1)); + c.getOrLoad(key("db", "t", 2L), () -> raws(2)); + c.getOrLoad(key("db", "other", 1L), () -> raws(3)); + Assertions.assertEquals(3, c.size()); + + // REFRESH TABLE db.t must drop BOTH snapshot entries of db.t and leave db.other intact. MUTATION: + // invalidateKey on a single (table, snapshot) key -> the other snapshot survives -> size 2 here -> red. + c.invalidate(TableIdentifier.of("db", "t")); + Assertions.assertEquals(1, c.size(), "only db.other's entry must survive"); + List reload = c.getOrLoad(key("db", "t", 1L), () -> { + loads.incrementAndGet(); + return raws(9); + }); + Assertions.assertEquals(9, reload.size()); + Assertions.assertEquals(1, loads.get()); + } + + @Test + public void invalidateDbClearsOnlyThatDbsTables() { + IcebergPartitionCache c = new IcebergPartitionCache(100, 1000); + c.getOrLoad(key("db1", "t1", 1L), () -> raws(1)); + c.getOrLoad(key("db1", "t2", 1L), () -> raws(2)); + c.getOrLoad(key("db2", "t1", 1L), () -> raws(3)); + Assertions.assertEquals(3, c.size()); + c.invalidateDb("db1"); + Assertions.assertEquals(1, c.size(), "only db2's entry must survive REFRESH DATABASE db1"); + } + + @Test + public void invalidateAllClearsEverything() { + IcebergPartitionCache c = new IcebergPartitionCache(100, 1000); + c.getOrLoad(key("db", "t1", 1L), () -> raws(1)); + c.getOrLoad(key("db", "t2", 1L), () -> raws(2)); + Assertions.assertEquals(2, c.size()); + c.invalidateAll(); + Assertions.assertEquals(0, c.size()); + } + + @Test + public void loaderExceptionPropagatesUnwrapped() { + // listPartitions catches ValidationException (dropped partition source column) to degrade to an empty + // list. Routing the scan through this cache must NOT wrap it, or the degradation would break. The + // MetaCacheEntry manual-miss-load path re-throws the loader's RuntimeException verbatim, and a failed + // scan is not cached. MUTATION: wrapping the loader exception -> assertThrows(ValidationException) fails. + IcebergPartitionCache c = new IcebergPartitionCache(100, 1000); + Assertions.assertThrows(ValidationException.class, () -> c.getOrLoad(key("db", "t", 5L), () -> { + throw new ValidationException("Cannot find source column for partition field"); + })); + Assertions.assertEquals(0, c.size(), "a failed scan must not be cached"); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergPartitionUtilsTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergPartitionUtilsTest.java index e41ab6246a69a0..cb42feafd95cb2 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergPartitionUtilsTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergPartitionUtilsTest.java @@ -824,6 +824,35 @@ public void buildMvccPartitionViewEnumeratesRangePartitions() { "a committed RANGE table must report a positive newest-update-time for dictionary refresh"); } + @Test + public void partitionScanIsCachedAcrossRepeatsAndConsumersAtSameSnapshot() { + // PERF-02 metric gate: buildMvccPartitionView (MTMV/RANGE) and listPartitions (selectedPartitionNum) both + // funnel through loadRawPartitions keyed by (table, currentSnapshotId). A shared IcebergPartitionCache must + // collapse repeated calls -- across queries AND across the two consumers at the same snapshot -- onto ONE + // remote PARTITIONS scan (restoring legacy cross-query partition-info caching + collapsing the MTMV 4~6x + // re-enumeration). MUTATION: not threading the cache into loadRawPartitions -> each call re-scans -> + // loadCountForTest > 1 -> red. + PartitionSpec spec = PartitionSpec.builderFor(RELATED_SCHEMA).day("ts").build(); + Table table = dayPartitionedTable(spec, "ts_day=1970-04-11", "ts_day=1970-07-20"); + TableIdentifier id = TableIdentifier.of("db1", "t"); + IcebergPartitionCache cache = new IcebergPartitionCache(100, 1000); + + ConnectorMvccPartitionView v1 = IcebergPartitionUtils.buildMvccPartitionView(table, -1L, id, cache); + IcebergPartitionUtils.buildMvccPartitionView(table, -1L, id, cache); + List parts = IcebergPartitionUtils.listPartitions(table, id, cache); + + Assertions.assertEquals(1, cache.loadCountForTest(), + "the PARTITIONS scan must run exactly once across repeated views + both consumers at one snapshot"); + Assertions.assertEquals(1, cache.size(), "one (table, snapshot) entry"); + Assertions.assertEquals(2, parts.size(), "listPartitions still returns the two physical partitions"); + // Parity: the cached view matches a fresh (uncached) enumeration. + List cachedNames = v1.getPartitions().stream() + .map(ConnectorMvccPartition::getName).collect(Collectors.toList()); + List liveNames = IcebergPartitionUtils.buildMvccPartitionView(table, -1L).getPartitions().stream() + .map(ConnectorMvccPartition::getName).collect(Collectors.toList()); + Assertions.assertEquals(liveNames, cachedNames, "cached partition view must equal a live enumeration"); + } + @Test public void buildMvccPartitionViewResolvesPerPartitionSnapshotId() { // Two SEPARATE commits: ts_day=100 lands in snapshot S1, ts_day=200 in S2. Each partition's freshness is From e1e6d0208752de496398694182688976b31f2692 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 01:48:40 +0800 Subject: [PATCH 07/47] [doc](catalog) fe-connector-iceberg PERF-02: design + summary + tracker updates; next = PERF-03 Record PERF-02 (518d0599cbf) as done: add the partition-view-cache design and summary, mark it complete in tasklist, append the session-2 progress entry, and repoint HANDOFF at PERF-03 (#64134 planFiles format-inference fallback). Notes the stale PERF-01 dependency (Part B was dropped) so PERF-03 targets the scan itself. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 48 +++++++++---------- ...FIX-PERF-02-partition-view-cache-design.md | 48 +++++++++++++++++++ ...IX-PERF-02-partition-view-cache-summary.md | 33 +++++++++++++ plan-doc/perf-hotpath-iceberg/progress.md | 11 +++++ plan-doc/perf-hotpath-iceberg/tasklist.md | 4 +- 5 files changed, 117 insertions(+), 27 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-02-partition-view-cache-design.md create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-02-partition-view-cache-summary.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index 701c2254b49c10..2b1b91f2278046 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -6,48 +6,46 @@ --- -# ✅ 上一个 session（session 2，2026-07-18）：**PERF-01 完成并全绿** +# ✅ 已完成（session 2，2026-07-18） -- **commit `484f0e0c125`**（`[perf]` 一个自包含 commit）。小结见 [`designs/FIX-PERF-01-table-memo-summary.md`](./designs/FIX-PERF-01-table-memo-summary.md)。 -- 落地 = **胖 handle（`IcebergTableHandle.transient resolvedTable`，查询内单实例）+ 跨查询 `IcebergTableCache`（挂 `IcebergConnector`，gate=`isUserSessionEnabled()||restVendedCredentialsEnabled()` 关）**。统一读 helper `IcebergConnectorMetadata.resolveTableForRead`（胖 handle→cache→裸 loadTable）；provider `resolveTable` 胖 handle 优先 + per-call `wrapTableForScan`。Part B 未做（红队证伪）。 -- **验证**：全 iceberg 模块 **932 pass / 0 fail / 1 skip**，checkstyle 绿。度量守门 `planningPassLoadsSameTableOnceViaFatHandle`（规划期同表远端 loadTable = 1，修前 2）。 -- **可复用产物**：`(TableIdentifier)` 键的 `IcebergTableCache` 基建 + 胖 handle 携带模式 + `RecordingIcebergCatalogOps` 计数守门法 —— PERF-02/03/10 直接复用，别再造。 +- **PERF-01**（commit `484f0e0c125`）：胖 handle（`IcebergTableHandle.transient resolvedTable`，查询内）+ 跨查询 `IcebergTableCache`（挂 `IcebergConnector`，凭证 gate）。同表规划期远端 `loadTable` 3~7→1。 +- **PERF-02**（commit `518d0599cbf`）：跨查询 `IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、值 raw 分区列表、**无凭证 gate**——纯元数据）。分区表分析期 PARTITIONS 扫描每查询→每快照一遍；MTMV 一次 refresh 的 4~6 遍→1。小结见 `designs/FIX-PERF-02-*-summary.md`。 +- 两轮均全 iceberg 模块单测绿（943/0/1），checkstyle 绿，0 回归。 +- **可复用产物**：`IcebergTableCache`(键 TableIdentifier) / `IcebergPartitionCache`(键 `(TableIdentifier,snapshotId)`+`loadCountForTest` 度量守门) 两套 `MetaCacheEntry` 基建；凭证 gate 判据（值含 FileIO→gate，纯元数据→无 gate）；胖 handle 携带模式。PERF-03/10 复用。 --- -# 🆕 下一个 session = **PERF-02（分区视图跨查询缓存 + MTMV refresh pin，C7 C22 C23）** +# 🆕 下一个 session = **PERF-03（#64134 planFiles 兜底复活，C2 C11）** ## 第一件事（立项流程见 README §单项立项流程） -1. **复核（动码前，行号信 grep 不信文档）**：按 findings.json C7/C22/C23 重新 grep：分析期 `loadSnapshot → materializeLatest` 对分区表走 `IcebergPartitionUtils.loadRawPartitions`（PARTITIONS 元数据表 `planFiles()+rows()`，读该快照全部 data+delete manifest）；`StatementContext` 只单语句 memoize、跨查询零缓存；MTMV 一次 refresh 里 `isValidRelatedTable/alignMvPartition/generateRelatedPartitionDescs/getAndCopyPartitionItems` 各 materialize 同视图 4~6 次（无 refresh 级 pin + 枚举点间快照偏移风险）。确认成本/乘数仍成立。 -2. **设计**（写 `designs/FIX-PERF-02-...-design.md`）：按 `(TableIdentifier, snapshotId)` 缓存**分区视图**（pin 已在 `beginQuerySnapshot` 后落在 handle 上），挂 `IcebergConnector`（复用 PERF-01 的 `MetaCacheEntry` 基建套路，值=分区视图对象）；MTMV 侧在 `MTMVRefreshContext` 加 refresh 级 `MvccSnapshot` pin。**注意 gate**：分区视图不携带凭证（是元数据），但仍要想清跨查询新鲜度语义是否 = legacy。 -3. **设计红队**（对抗 review，clean-room 偏好）后再实现。 -4. **TDD**：先 `RecordingIcebergCatalogOps`/分区扫描计数守门（分析期同表分区视图 build 从 N→1）。 -5. 参考 PERF-01 的 gate/失效/handle-pin 三板斧，别各造一套 key。 +1. **复核（动码前，行号信 grep 不信文档）**：按 findings.json C2/C11 重新 grep：`getScanNodeProperties → IcebergWriterHelper.getFileFormat → resolveFileFormatName → inferFileFormatFromDataFiles → table.newScan().planFiles()`（无过滤**整表 manifest 扫描**，只为拿"第一个数据文件的格式"）。门槛 = 表属性无 `write-format` 且无 `write.format.default`（迁移表 + 任何写引擎从不显式设该属性的表）。乘数 = 每次 properties 计算 1 次（无谓词 1/查询、带谓词 2/查询）。确认成本/乘数/纯冗余（planScan 自己的 planFiles 枚举里每个 `dataFile.format()` 都有）。 +2. **⚠ 依赖已过时**：tasklist 写"依赖 PERF-01 的 convertPredicate 收窄消第二次"——但 **PERF-01 已删 Part B（convertPredicate 收窄，红队证伪为 no-op）**，故不存在"第二次"可消；PERF-03 就是把这一次整表扫描本身干掉。 +3. **设计**（写 `designs/FIX-PERF-03-...-design.md`）：两条路线择一或结合——(a) 按 `(table UUID/TableIdentifier, snapshotId)` memoize 解析出的 format（快照不变⇒格式不变，复用 PERF-02 的 `(table,snapshotId)` 键+cache 套路，值=一个 `FileFormat`/字符串，纯元数据无 gate）；或 **(b) 从 planScan 自己的 split 枚举结果反推 scan 级 format 传下去**（更彻底：连那"一次"整表扫描都不做，因为数据规划本就要枚举 dataFile）。**倾向 (b)**（消除而非缓存重操作，更符合问题类），但要核 `getScanNodeProperties`(属性路径) 与 `planScan`(数据路径) 的先后/是否同 handle 血缘能共享枚举结果——若两路径独立触发则 (b) 需跨路径传递，退回 (a) memoize。 +4. **红队 + TDD**：度量守门 = 无 write-format 的表规划期 `planFiles`（或 format 解析）远端次数从 1→0（(b)）或跨查询 1→命中（(a)）。可仿 PERF-02 的 `loadCountForTest` 计数法。 +5. 守铁律：memo/派生放连接器侧；fe-core 不解析属性（format 组装在插件侧→thrift）。 -## ⚠️ 关键认知（承 PERF-01，别重踩） +## ⚠️ 关键认知（承 PERF-01/02） -- **胖 handle memo 严格查询内**：`PluginDrivenScanNode.currentHandle` 每查询新建、`resolveConnectorTableHandle`→每次 fresh `getTableHandle`、不挂长生命周期 `ExternalTable`。任何"跨查询"缓存必须挂 `IcebergConnector`（长生命周期），不能靠 handle。 -- **凭证红线**：任何缓存**携带 FileIO 凭证的对象**（raw Table）都要 gate 掉 `session=user`/`vended`；纯元数据（快照 id、分区视图值）不需要，但要确认值里不夹带 FileIO。 -- **测试共享 static handle 会串味**：胖 handle 让 handle 带可变状态，任何跨用例复用的 `static final IcebergTableHandle` 都要改成每用例 fresh（PERF-01 踩过 `T1`）。 +- **缓存都挂 `IcebergConnector`**（长生命周期）；键统一用 `(TableIdentifier[, snapshotId])`，snapshotId 由 `IcebergLatestSnapshotCache` 稳定。别靠 handle 跨查询（handle 每查询新建）。 +- **凭证 gate 判据**：缓存值**含 FileIO/凭证**（如 raw Table）→ gate 掉 `session=user`/`vended`；**纯元数据**（快照 id、分区列表、format）→ 无 gate。PERF-03 的 format 是纯元数据 → 无 gate。 +- **测试禁共享可变 handle/静态态**：胖 handle 后 handle 带可变状态，跨用例复用的 `static` handle 会串味（PERF-01 踩过 `T1`）。 +- **scan 级 format_type 决定 V1/V2 scanner**（见全局 memory）：PERF-03 若走 (b) 反推 format 传下去，务必别把整个连接器错钉在 `file_format_type=jni`（只 isSystemTable 该发 jni），对齐上游同名 ScanNode 的 `getFileFormatType()`。 --- -# 🧰 构建/验证坑（**本轮实证更新，务必照做**） +# 🧰 构建/验证坑（**实证，务必照做**） -1. **可靠跑单测 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`**（绝对 `-f /fe/pom.xml`）。原因： - - 本 worktree 用 `${revision}` CI 版本，已装的子模块 pom **未 flatten**（parent 仍写 `${revision}`）→ `-pl iceberg` **单模块**永远解析不到 `fe-connector:pom:${revision}`（且 `-Drevision=` 不透传到传递依赖 pom）→ 必须 `-am`（reactor 解析 revision）。 - - `-am` + `test` 相**只到 test**，不产 `fe-connector-hms-hive-shade` 的 shade jar → iceberg 编译报 `HiveConf` 找不到；故用 **`install`**（到 package 相跑 shade）。 - - `-Dtest=` 让**上游模块 0 匹配测试**快速跳过，只 iceberg 跑全套（列表 = `ls src/test/.../*Test.java`）。 -2. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 被静默跳过，BUILD SUCCESS 是陈旧文件）。 -3. **后台 task 通知的 "exit code" 是尾部 `tail`/`echo` 的**，非 maven —— 读日志的 `BUILD SUCCESS`/`Tests run:` 行。 -4. **checkstyle**：LineLength(120) 对 **test 源已 suppress**；主源须 ≤120；import 序 = `SAME_PACKAGE(3)→THIRD_PARTY→STANDARD_JAVA` 组内字母序、组间空行。单独跑 `mvn checkstyle:check -pl ... -am`。 +1. **可靠跑单测 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`**（绝对 `-f /fe/pom.xml`）。原因：本 worktree `${revision}` CI 版本 + 未 flatten 的已装 pom → `-pl iceberg` 单模块解析不到 `fe-connector:pom:${revision}`（`-Drevision=` 不透传传递依赖），必须 `-am`；且 `-am test` 只到 test 相不产 hms-hive-shade 的 shade jar（缺 `HiveConf`），故用 **`install`**（到 package 相跑 shade）。`-Dtest=`（=`ls src/test/.../*Test.java`）让上游 0 匹配测试快跳。**checkstyle 单独跑** `mvn checkstyle:check -pl ... -am`（测试跑时 `-Dcheckstyle.skip=true`）。 +2. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 静默跳过）。 +3. **后台 task 通知的 "exit code" 是尾部命令的**，非 maven —— 读日志 `BUILD SUCCESS`/`Tests run:` 行（`Monitor` 用 `until grep -qE 'BUILD SUCCESS|BUILD FAILURE'` 守 5s 轮询）。 +4. **checkstyle**：LineLength(120) 对 **test 源已 suppress**；主源 ≤120；import 序 `SAME_PACKAGE(3)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行（`org.apache.doris.*`=SAME_PACKAGE，含嵌套类 import 不算 redundant）。 5. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。 -6. **并发 session 共享同一 worktree** —— 动码前探测（`git log/status` + `pgrep maven` 看 `etime`）。 +6. **并发探测**：`pgrep -a -f maven`。注意 `/mnt/disk1/yy/git/doris`（另一 worktree）常有构建在跑——那不动本 `wt-catalog-spi` 的文件，非冲突；只看**本目录** `find fe -newermt '-120 sec'`。 7. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 --- # 🗂 开放问题 -- 无。PERF-01 已合，进入 PERF-02。 +- 无。PERF-01/02 已合，进入 PERF-03。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-02-partition-view-cache-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-02-partition-view-cache-design.md new file mode 100644 index 00000000000000..e908858f9e8e42 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-02-partition-view-cache-design.md @@ -0,0 +1,48 @@ +# FIX-PERF-02 — 分区视图跨查询缓存（(table, snapshotId) → raw partitions） + +> 覆盖审计发现 C7 C23 C22（簇3）。基线 HEAD `65cc6d4da2d`（PERF-01 已合），分支 `catalog-spi-review-16`。 +> 行号信 grep 不信本文档。决策已定（用户 2026-07-18）：**连接器侧 `(TableIdentifier, snapshotId)` 缓存 raw partitions；不动 fe-core、不加 MTMV refresh pin（见 §5.3）。** + +--- + +## 1. Problem + +对分区表，每条查询在**分析期**（`BindRelation → StatementContext.loadSnapshots → PluginDrivenMvccExternalTable.materializeLatest`）会走 `getMvccPartitionView`（RANGE/time-transform 表）或 `listPartitions`（其余分区表）→ `IcebergPartitionUtils.loadRawPartitions` = 扫 iceberg PARTITIONS 元数据表（`partitionsTable.newScan().useSnapshot(id).planFiles() + rows()`，SDK 聚合时读该快照**全部** data+delete manifest，O(#manifests) 远程 IO，**与查询选择性无关**）。`StatementContext` 只单语句 memoize，跨查询零缓存。SHOW PARTITIONS（`listPartitionNames`）与 selectedPartitionNum（`listPartitions`）同源。MTMV 一次 refresh 里同一视图被 materialize **4~6 次**。 + +## 2. Root Cause（含复核修正） + +- SPI 迁移把 legacy `IcebergExternalMetaCache.loadSnapshotProjection`（把 `loadPartitionInfo` 缓进 TTL'd tableEntry）拆掉；`IcebergLatestSnapshotCache` 只存 `(snapshotId,schemaId)`，**不缓存派生的分区视图**。故对 MTMV-eligible 表这是**丢失的缓存回退**；对非 eligible 分区表，per-query PARTITIONS 扫描是 selectedPartitionNum 特性引入的**新增未缓存成本**（复核实证 vs master）。 +- **PERF-01 已消除本簇的 `loadTable` 那半**（三处已走 `resolveTableForRead`）；**剩余未缓存 = `loadRawPartitions` 的 PARTITIONS 扫描**。本任务只补这半。 +- **快照键可用**：三消费方内部都把 snapshotId 解析成**具体值**（`buildMvccPartitionView`：pin>=0 用 pin，否则 `currentSnapshot()`；`listPartitions`/`listPartitionNames`：`currentSnapshot()`），故缓存须在 `IcebergPartitionUtils` 内（解析点）按解析后的 snapshotId 建键。该 snapshotId 已由 `IcebergLatestSnapshotCache` 稳定（同 TTL 内不变）。 +- **MTMV 放大自动收敛**：4~6 次 materialize 各经 `beginQuerySnapshot → IcebergLatestSnapshotCache`（同一 refresh 内命中同一 snapshotId）→ 同一 `(id, snapshotId)` 键 → 本缓存命中，坍缩重复远程扫描（无需 refresh pin，见 §5.3）。 + +## 3. Design（连接器侧，零 fe-core 改动） + +**新增 `IcebergPartitionCache`（挂 `IcebergConnector`）**：`MetaCacheEntry>`，`Key=(TableIdentifier id, long snapshotId)`。值 = `loadRawPartitions` 的原始分区列表（纯元数据：名/值/transform/时间戳/快照 id，**无 FileIO/凭证**）。TTL = 同一 `meta.cache.iceberg.table.ttl-second`，capacity 复用默认；manual-miss-load（异常原样透传，保 `listPartitions` 的 `ValidationException`→空降级；失败不入缓存）。 + +- **无凭证 gate**（≠ PERF-01 的 `IcebergTableCache`）：值不含凭证，跨用户共享安全 → 对所有目录无条件构造。 +- **消费**：`IcebergPartitionUtils.loadRawPartitions` 先查缓存；`buildMvccPartitionView`/`listPartitions`/`listPartitionNames` 三方法加 `(TableIdentifier id, IcebergPartitionCache cache)` 参（旧签名保留为 null 重载给单测）。三方法共享同一份 raw → SHOW PARTITIONS / selectedPartitionNum / MTMV 视图互相命中。 +- **值不可变共享**：`IcebergRawPartition` 全 final（改为包内可见供缓存引用）；缓存的列表 `unmodifiableList` 包一层，跨并发只读安全。 +- **失效**：`IcebergConnector.invalidate{Table,Db,All}` 各加 `partitionCache` 清空（`invalidate(id)` 用 `invalidateIf(k -> k.id.equals(id))` 清该表所有快照条目；`invalidateDb` 按 namespace；`invalidateAll` 全清）。 +- **快照/schema 可见性 parity**：同 `(id, snapshotId)` 永远同结果（快照不可变）；跨快照（新提交→新 snapshotId→新键）miss→实扫。= master TTL 语义，非新增 staleness。 + +## 4. Implementation Plan（一个 commit，iceberg 自包含） + +1. `IcebergPartitionUtils`：`IcebergRawPartition` `private`→包内；`loadRawPartitions` 拆 `loadRawPartitionsUncached` + cache-aware 包装；三 public 方法加 `(id, cache)` 参 + 旧签名重载（null,null）。 +2. 新增 `IcebergPartitionCache.java`（+ 静态 `Key`；`getOrLoad/invalidate/invalidateDb/invalidateAll/isEnabled/size/loadCountForTest`）。 +3. `IcebergConnector`：构造 `partitionCache`（无 gate）；`getMetadata` 传入；`invalidate*` 三钩子；`partitionCacheForTest()`。 +4. `IcebergConnectorMetadata`：ctor 加 `partitionCache` 参（便利 ctor 传 null）；`getMvccPartitionView`/`listPartitionNames`/`listPartitions` 三处传 `(TableIdentifier.of(db,tbl), partitionCache)`。 + +## 5. Risks & Open Questions + +1. **凭证**：raw partitions 无凭证 → 无 gate（已核 `IcebergRawPartition` 字段）。 +2. **DDL 隔离**：DDL/写不走分区枚举路径，天然隔离；REFRESH 清缓存。 +3. **MTMV refresh pin 不做（范围决定，用户 2026-07-18 认可）**：ttl>0（默认）时 `IcebergLatestSnapshotCache` 已稳定住一次 refresh 内的 snapshotId，本缓存据此坍缩 4~6 次重复 → 性能目标达成，无需 fe-core 的 refresh 级 pin。仅 `ttl=0`（无缓存目录，用户主动放弃缓存）残留"枚举点间快照偏移"的**既有**一致性边角（重构前即存在），修它须改 fe-core 物化视图刷新上下文，违反"缓存放连接器、fe-core 只出不进"铁律 → 标为超范围，如需单开框架侧任务。 +4. **CPU 派生未缓存**：缓存的是 raw（远程 IO 那半，审计的 heavy op）；`buildMvccPartitionView` 的 mergeOverlap 等 CPU 派生每次重跑（微秒~毫秒级，非远程 IO）。可接受；如成瓶颈另立缓存 final view 的任务。 + +## 6. Test Plan / 度量守门 + +- **缓存单测** `IcebergPartitionCacheTest`：计数 loader → TTL 内命中 loader 跑 1 次；`ttl<=0`/负值关；invalidate/invalidateDb/invalidateAll；异常原样透传。 +- **度量守门（集成）**：真 InMemory 分区表 + 真 cache，`buildMvccPartitionView`（及 `listPartitions`）同 `(id, snapshotId)` 调两次 → `cache.loadCountForTest() == 1`（远程分区扫描跨查询恰 1 次；修前 = 2）。 +- **parity**：现有 `IcebergPartitionUtilsTest`/MVCC/分区计数单测全绿。 +- **build**：`install -am` + iceberg 测试类过滤 + `-Dmaven.build.cache.enabled=false`。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-02-partition-view-cache-summary.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-02-partition-view-cache-summary.md new file mode 100644 index 00000000000000..5726f37d9b7e09 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-02-partition-view-cache-summary.md @@ -0,0 +1,33 @@ +# FIX-PERF-02 — Summary（分区视图跨查询缓存） + +> 权威设计见 [`FIX-PERF-02-partition-view-cache-design.md`](./FIX-PERF-02-partition-view-cache-design.md)。本文件只记落地结果。 +> commit `518d0599cbf`（`[perf]`，iceberg 自包含）。基线 HEAD `65cc6d4da2d`（PERF-01 已合）。 + +## Problem + +分区表每条查询在分析期扫一遍 PARTITIONS 元数据表（`loadRawPartitions` → SDK 读该快照全部 data+delete manifest，O(#manifests) 远程 IO，与查询选择性无关），只单语句 memoize、跨查询零缓存；MTMV 一次 refresh 里同一视图被 materialize 4~6 次。= 重构时丢掉的 legacy `IcebergExternalMetaCache` 分区信息缓存。PERF-01 已消掉本簇的 `loadTable` 那半，本任务补 PARTITIONS 扫描这半。 + +## Fix（连接器侧，零 fe-core 改动） + +- **新增 `IcebergPartitionCache`**（挂 `IcebergConnector`）：键 `(TableIdentifier, snapshotId)`，值 = raw 分区列表。三消费方（`buildMvccPartitionView` / `listPartitions` / `listPartitionNames`）都经 `loadRawPartitions` → **共享同一份扫描**。快照 id 由上一轮 `IcebergLatestSnapshotCache` 稳定 → 键跨查询/跨一次 refresh 稳定 → MTMV 4~6 次也坍缩成 1 次。 +- **无凭证 gate**（≠ PERF-01 的 table cache）：值是纯元数据（名/值/transform/时间戳/快照 id），不含 FileIO/凭证 → 对所有目录无条件构造，仅 TTL 控制。 +- 快照不可变 → 同键永远同结果（parity）；新提交→新 snapshotId→新键→实扫。缓存值 `unmodifiableList`；loader 异常原样透传（保 `listPartitions` 的 `ValidationException`→空降级），失败不入缓存。 +- `IcebergPartitionUtils`：三方法加 `(TableIdentifier, IcebergPartitionCache)` 参 + 旧签名重载；`loadRawPartitions` 拆 cache-aware 包装 + `loadRawPartitionsUncached`；`IcebergRawPartition` 改包内可见。 +- `IcebergConnector`：构造 `partitionCache`（无 gate）、传入 metadata、三个 `invalidate*` 清空、`partitionCacheForTest()`。 +- `IcebergConnectorMetadata`：ctor 加 `partitionCache` 参（便利 ctor 传 null）；三处传 `(TableIdentifier.of(db,tbl), partitionCache)`。 + +## 范围决定（用户 2026-07-18 认可） + +审计另建议的 **MTMV refresh 级 MvccSnapshot pin 未做**：`ttl>0`（默认）时 `IcebergLatestSnapshotCache` 已稳定住一次 refresh 内的 snapshotId，本缓存据此坍缩 4~6 次重复 → 性能目标达成，无需改 fe-core。仅 `ttl=0`（无缓存目录）残留"枚举点间快照偏移"的**既有**一致性边角，修它须改 fe-core → 超范围。 + +## Tests + +- **缓存单测** `IcebergPartitionCacheTest`：TTL 内命中同实例 + loadCount=1、不同 snapshotId 不同键、`ttl<=0`/负值关、invalidate（清该表所有快照）/invalidateDb/invalidateAll、`ValidationException` 原样透传+失败不缓存。 +- **度量守门（集成）** `IcebergPartitionUtilsTest.partitionScanIsCachedAcrossRepeatsAndConsumersAtSameSnapshot`：真分区表 + 真 cache，重复 `buildMvccPartitionView` + `listPartitions` 同快照 → `loadCountForTest()==1`（远程分区扫描跨查询/跨消费方恰 1 次；修前每次一扫），并与无缓存枚举 parity。 +- **连接器 gate + 失效** `IcebergConnectorCacheTest`：plain/vended/session=user 三目录 partitionCache 均非空（无 gate）；REFRESH TABLE/DB/CATALOG 逐级清空。 + +## Result + +- 全 iceberg 模块 **943 pass / 0 fail / 1 skip**（`install -am`），checkstyle 绿。 +- 减负：分区表每查询分析期 PARTITIONS 远程扫描从"每次一遍"→"每快照一遍"（跨查询命中）；MTMV 一次 refresh 从 4~6 遍 → 1 遍。metastore/对象存储分区 IO 除以 QPS-over-TTL 与 refresh 枚举点数。 +- parity：缓存枚举 == 实时枚举；快照/schema 可见性、时间旅行取快照不变。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index 2c46d6aef52eab..3fd6690d47d19a 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -34,3 +34,14 @@ - **结果**：全 iceberg 模块 **932 pass / 0 fail / 1 skip**（`install -am`），checkstyle 绿。summary 见 `designs/FIX-PERF-01-table-memo-summary.md`。 - **构建坑记录**：本 worktree `${revision}` CI 版本 + 未 flatten 的已装 pom → `-pl iceberg` 单模块永远解析不到 `fe-connector:pom:${revision}`（`-Drevision=` 不透传到传递依赖 pom）；且 `-am test` 只到 test 相不产 hms-hive-shade 的 shade jar（缺 `HiveConf`）。**可靠跑法 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest= -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`**（reactor 解析 revision + 到 install 相产 shade jar + 上游 0 匹配测试快速跳过）。 - **下一步**：见 HANDOFF —— PERF-02（分区视图跨查询缓存 + MTMV refresh pin，复用本任务的 `(table, snapshotId)` pin 与胖 handle 模式）。 + +--- + +## 2026-07-18 — session 2（续）：PERF-02 实现 + 全绿（commit `518d0599cbf`） + +- **复核确认审计簇3**：分区表分析期 `materializeLatest → getMvccPartitionView/listPartitions → IcebergPartitionUtils.loadRawPartitions`（PARTITIONS 元数据表扫描，读该快照全部 manifest）跨查询零缓存；三消费方（MVCC 视图 / SHOW PARTITIONS / selectedPartitionNum）同源。**关键复核发现**：PERF-01 已消掉本簇 `loadTable` 那半，剩余 = PARTITIONS 扫描；且三方法内部才把 snapshotId 解析成具体值 → 缓存必须在 `IcebergPartitionUtils` 内按解析后 snapshotId 建键（非 metadata 层）。**vs master**：master 把 loadPartitionInfo 缓在 TTL'd tableEntry，故这是重构丢失的回退（非 eligible 分区表则是 selectedPartitionNum 特性引入的新增未缓存成本）。 +- **实现（连接器侧，一个 `[perf]` commit）**：新增 `IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、值 raw 分区列表、**无凭证 gate**——纯元数据）；`IcebergPartitionUtils` 三方法加 `(id, cache)` 参 + 旧签名重载、`loadRawPartitions` 拆 cache-aware + uncached、`IcebergRawPartition` 改包内可见；`IcebergConnector` 无条件构造 + 传入 + 三失效钩子 + test 访问器；`IcebergConnectorMetadata` ctor 加参 + 三处传 `(TableIdentifier, cache)`。 +- **范围决定（用户认可）**：**MTMV refresh 级 pin 不做**——`ttl>0` 时 `IcebergLatestSnapshotCache` 已稳定一次 refresh 内 snapshotId，本缓存据此把 4~6 次重复坍缩成 1，无需改 fe-core；仅 `ttl=0` 无缓存目录残留既有枚举偏移边角，超范围。 +- **度量守门（新）**：`IcebergPartitionUtilsTest.partitionScanIsCachedAcrossRepeatsAndConsumersAtSameSnapshot` —— 真分区表重复 `buildMvccPartitionView` + `listPartitions` 同快照 → `loadCountForTest()==1`。另 `IcebergPartitionCacheTest`（含 `ValidationException` 透传+不缓存）、连接器 gate/失效诸测。 +- **结果**：全 iceberg 模块 **943 pass / 0 fail / 1 skip**，checkstyle 绿。summary 见 `designs/FIX-PERF-02-partition-view-cache-summary.md`。0 回归（无需改任何现有测试）。 +- **下一步**：见 HANDOFF —— PERF-03（#64134 planFiles 兜底复活：`file_format_type` 无 write-format 时走整表 planFiles 反推格式 → memoize / 从枚举反推）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index b60e971b84974b..b56adadc27047f 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -16,7 +16,7 @@ | ID | 优先级 | 覆盖发现 | 主题（一句话） | 依赖 | 状态 | commit | |---|---|---|---|---|---|---| | PERF-01 | P0 | C1 C4 C6 C10 C16 | 一次规划 3~7 次远程 loadTable → 胖 handle(查询内单实例)+ 跨查询 IcebergTableCache(挂 Connector);~~convertPredicate 收窄~~已删(红队证伪) | — | ✅ 完成 | `484f0e0c125` | -| PERF-02 | P0 | C7 C22 C23 | 分区视图每查询重扫 PARTITIONS 元数据表 → `(table,snapshotId)` 缓存 + MTMV refresh pin | 与 01 共享快照 pin 机制 | ⏳ | | +| PERF-02 | P0 | C7 C22 C23 | 分区视图每查询重扫 PARTITIONS 元数据表 → `(table,snapshotId)` 缓存(连接器侧,无凭证 gate);MTMV refresh pin 判定为多余(靠 latestSnapshotCache 稳定快照坍缩,不改 fe-core) | 与 01 共享快照 pin 机制 | ✅ 完成 | `518d0599cbf` | | PERF-03 | P0 | C2 C11 | #64134 复活：`file_format_type` 兜底走整表 planFiles → memoize / 从枚举反推 | 受益于 01 的失效收窄（消第二次） | ⏳ | | | PERF-04 | P1 | C17 C18 | streaming / COUNT(*) 下推旁路 IcebergManifestCache → 两条旁路接回 | — | ⏳ | | | PERF-05 | P1 | C9 | information_schema.tables 循环内每表 loadTable（只为拿 comment） → 随表缓存 / 惰性取 | — | ⏳ | | @@ -38,7 +38,7 @@ - **约束**：memo 放**连接器侧**，**不得**在 fe-core 加 Table 缓存或属性派生（见 README 铁律）。 - **依赖**：无。**建议第一个做** —— 收益最大，且其失效收窄是 PERF-03、PERF-10 的前置。 -### [ ] PERF-02 — 簇3：分区视图跨查询缓存 + MTMV refresh pin（C7 C22 C23） +### [x] PERF-02 — 簇3：分区视图跨查询缓存（C7 C22 C23） · ✅ `518d0599cbf` - **病灶**：分析期 `loadSnapshot → materializeLatest:126` 对分区表走 `IcebergPartitionUtils.loadRawPartitions:709-718` = PARTITIONS 元数据表 `planFiles()+rows()`（SDK 聚合读该快照**全部** data+delete manifest）。`StatementContext` 只在单语句内 memoize，跨查询零缓存（legacy 二级分区缓存在 CACHE-P1 决策中被有意放弃）。MTMV 放大（C22）：一次 refresh 里 `isValidRelatedTable/alignMvPartition/generateRelatedPartitionDescs/getAndCopyPartitionItems` 各 materialize 同视图 **4~6 次**（无 refresh 级 pin，且引入枚举点间快照偏移风险）。 - **修复方向**：按 `(TableIdentifier, snapshotId)` 缓存分区视图（pin 在 `beginQuerySnapshot` 后已在 handle 上），挂 `fe-connector-cache`；MTMV 侧在 `MTMVRefreshContext` 加 refresh 级 `MvccSnapshot` pin。 - **依赖**：与 PERF-01 共享 `(table, snapshotId)` 快照 pin —— 先做 01 立住模式，02 复用。 From 7ab667c03f9eb4167153014ae39909f510f9738b Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 07:22:17 +0800 Subject: [PATCH 08/47] [perf](catalog) fe-connector-iceberg: memoize the file-format inference across queries (PERF-03) Root cause: every SELECT/EXPLAIN of an iceberg table resolves the scan-level file_format_type through IcebergScanPlanProvider.getScanNodeProperties -> IcebergWriterHelper.getFileFormat. When the table carries neither the 'write-format' nickname nor 'write.format.default' (migrated tables, and any table whose writer never set the property -- parquet is an implicit default, not stored), resolution falls back to inferFileFormatFromDataFiles, an unfiltered table.newScan().planFiles() -- a whole-table manifest scan (remote FileIO; iceberg's ParallelIterable eagerly fans manifest readers) just to read the first data file's format. That #64134 fallback ran on every query with zero cross-query reuse: hundreds of ms to seconds of remote IO per query on large tables, multiplied by QPS. Fix (connector-side, no fe-core change): add IcebergFormatCache keyed by (TableIdentifier, currentSnapshotId), value = the inferred format-name string, owned by the long-lived IcebergConnector. A snapshot is immutable, so its first data file's format is a pure function of the key; the snapshot id is held stable across queries by IcebergLatestSnapshotCache. Only the inference fallback is cached -- the two cheap property probes stay uncached. No credential gate (like the PERF-02 partition cache): the value is bare metadata with no FileIO/token, so it is shared across users and built for every catalog (incl. session=user / REST vended). REFRESH TABLE/DB/CATALOG invalidate it. Route note: deriving the format from planScan's own file enumeration was rejected -- FileQueryScanNode.createScanRangeLocations computes file_format_type (getFileFormatType, line 325) BEFORE getSplits/planScan (line 422), and the inference is unfiltered while planScan is predicate-filtered, so a query that prunes all files would break the legacy "unfiltered fallback always finds a first file" parity. Memoization keeps strict parity. Parity: same currentSnapshot-derived inference as legacy IcebergUtils.getFileFormat (never the handle's time-travel pin); the orc/parquet mapping (and its throw on an unsupported format) stays OUTSIDE the cache, so an unsupported first file is inferred once yet re-throws every call as before. A failed inference is NOT cached (the loader propagates unchecked; the CloseableIterable.close IOException is wrapped UncheckedIOException), so a transient remote-IO failure retries on the next query. The scan-level value still only selects BE's V1/V2 scanner; per-file format still travels per range. Write-path getFileFormat callers are unchanged (left for PERF-07). Measurement gate: IcebergWriterHelperTest asserts the whole-table inference runs exactly once across repeated queries at one snapshot (loadCountForTest == 1) and matches a live resolution; property-path tables never populate the cache. IcebergFormatCacheTest covers TTL stability, the ttl<=0 disable, invalidation, and not-caching-on-failure. IcebergConnectorCacheTest asserts the no-gate build across plain/vended/session catalogs and REFRESH invalidation. Full iceberg module UT: 957 run / 0 fail / 0 error / 1 skip; checkstyle green. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../connector/iceberg/IcebergConnector.java | 17 +- .../connector/iceberg/IcebergFormatCache.java | 145 ++++++++++++++ .../iceberg/IcebergScanPlanProvider.java | 26 ++- .../iceberg/IcebergWriterHelper.java | 79 +++++++- .../iceberg/IcebergConnectorCacheTest.java | 52 +++++ .../iceberg/IcebergFormatCacheTest.java | 181 ++++++++++++++++++ .../iceberg/IcebergWriterHelperTest.java | 81 ++++++++ 7 files changed, 573 insertions(+), 8 deletions(-) create mode 100644 fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergFormatCacheTest.java diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java index 6a86035c6795f2..88a0a788a2a98b 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java @@ -168,6 +168,10 @@ public class IcebergConnector implements Connector { // Built unconditionally — the cached value is pure metadata with no FileIO/credential, so no credential gate // (unlike tableCache); only the TTL knob disables it. private final IcebergPartitionCache partitionCache; + // PERF-03: cross-query inferred-file-format cache (the whole-table planFiles() fallback result, keyed by + // (table, snapshotId)). Like partitionCache, the value is pure metadata (a format-name string) with no + // FileIO/credential, so no gate; only the TTL knob disables it. + private final IcebergFormatCache formatCache; private final IcebergManifestCache manifestCache = new IcebergManifestCache(); // commit-bridge supply (S4 part 2): per-catalog stash carrying a row-level DML's non-equality delete supply // across the scan->write seam — the scan provider fills it (keyed by queryId), the write provider drains it @@ -212,6 +216,9 @@ public IcebergConnector(Map properties, ConnectorContext context // PERF-02: partition-view cache. Pure metadata (no credentials) -> no gate; same TTL/capacity as above. this.partitionCache = new IcebergPartitionCache( resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY); + // PERF-03: inferred-file-format cache. Pure metadata (no credentials) -> no gate; same TTL/capacity. + this.formatCache = new IcebergFormatCache( + resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY); } /** @@ -551,6 +558,7 @@ public void invalidateTable(String dbName, String tableName) { tableCache.invalidate(TableIdentifier.of(dbName, tableName)); } partitionCache.invalidate(TableIdentifier.of(dbName, tableName)); + formatCache.invalidate(TableIdentifier.of(dbName, tableName)); } /** @@ -571,6 +579,7 @@ public void invalidateDb(String dbName) { tableCache.invalidateDb(dbName); } partitionCache.invalidateDb(dbName); + formatCache.invalidateDb(dbName); } /** @@ -587,6 +596,7 @@ public void invalidateAll() { tableCache.invalidateAll(); } partitionCache.invalidateAll(); + formatCache.invalidateAll(); manifestCache.invalidateAll(); } @@ -626,6 +636,11 @@ IcebergPartitionCache partitionCacheForTest() { return partitionCache; } + /** Test-only: the cross-query inferred-file-format cache (PERF-03; always built, no credential gate). */ + IcebergFormatCache formatCacheForTest() { + return formatCache; + } + @Override public ConnectorScanPlanProvider getScanPlanProvider() { // Mirrors PaimonConnector.getScanPlanProvider: build a fresh provider per call over the lazily-built @@ -635,7 +650,7 @@ public ConnectorScanPlanProvider getScanPlanProvider() { // threaded for parity with the legacy single per-catalog IcebergMetadataOps. return new IcebergScanPlanProvider(properties, this::newCatalogBackedOps, context, manifestCache, - rewritableDeleteStash, tableCache); + rewritableDeleteStash, tableCache, formatCache); } @Override diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java new file mode 100644 index 00000000000000..294b29b3738e8c --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergFormatCache.java @@ -0,0 +1,145 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.cache.CacheSpec; +import org.apache.doris.connector.cache.MetaCacheEntry; + +import org.apache.iceberg.catalog.Namespace; +import org.apache.iceberg.catalog.TableIdentifier; + +import java.util.Objects; +import java.util.concurrent.ForkJoinPool; +import java.util.function.Supplier; + +/** + * Per-catalog cache of an iceberg table's inferred write file-format name (PERF-03), keyed by + * {@code (TableIdentifier, snapshotId)}. Kills the per-query revival of the #64134 heavy fallback: when a table + * carries neither the {@code write-format} nickname nor {@code write.format.default} (migrated tables, and any + * table whose writer never set the property — parquet is an implicit default, not stored), + * {@link IcebergScanPlanProvider#getScanNodeProperties} resolves the scan-level {@code file_format_type} through + * {@link IcebergWriterHelper#getFileFormat}, which falls back to an unfiltered {@code table.newScan().planFiles()} + * — a whole-table manifest scan (remote FileIO) — just to read the FIRST data file's format. That fallback was + * re-run on every query/EXPLAIN with no cross-query reuse; this cache collapses it to one scan per + * {@code (table, snapshot)}. + * + *

Snapshot-keyed, so always correct. A snapshot is immutable, so its first data file's format is a + * pure function of the key; a new commit yields a new snapshot id (a new key -> a live scan). Within the TTL + * the snapshot id itself is held stable by {@link IcebergLatestSnapshotCache}, which is what makes the key stable + * across queries. The key snapshot is the table's {@code currentSnapshot().snapshotId()} — exactly what the + * inference reads (it scans {@code table.newScan()}, never the handle's time-travel pin), matching legacy + * {@code IcebergUtils.getFileFormat} and PERF-02's partition cache verbatim. + * + *

Only the inference fallback is cached. The two cheap property probes ({@code write-format}, + * {@code write.format.default}) stay outside this cache — the caller consults it only when both miss. + * + *

No credential gate (unlike {@link IcebergTableCache}, like {@link IcebergPartitionCache}): the cached + * value is a bare format-name {@link String} with no {@code FileIO} / credential, so it is safe to share across + * users and is built unconditionally (only the TTL knob disables it). + * + *

Backed identically to {@link IcebergPartitionCache}: a contextual, access-TTL {@link MetaCacheEntry} with + * manual miss-load, so the inference runs OUTSIDE Caffeine's compute lock and a failed scan's exception + * propagates verbatim and is NOT cached (the next query retries — legacy parity). TTL is + * {@code meta.cache.iceberg.table.ttl-second}; {@code <= 0} disables (read live). Lives on the long-lived + * per-catalog {@link IcebergConnector}; a REFRESH CATALOG rebuilds the connector and thus the cache. + */ +final class IcebergFormatCache { + + /** Immutable composite key: a table's inferred format is distinct per pinned snapshot id. */ + static final class Key { + final TableIdentifier id; + final long snapshotId; + + Key(TableIdentifier id, long snapshotId) { + this.id = id; + this.snapshotId = snapshotId; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof Key)) { + return false; + } + Key that = (Key) o; + return snapshotId == that.snapshotId && Objects.equals(id, that.id); + } + + @Override + public int hashCode() { + return Objects.hash(id, snapshotId); + } + } + + private final MetaCacheEntry entry; + + IcebergFormatCache(long ttlSeconds, int maxSize) { + // Mirror IcebergPartitionCache: translate the connector's "<= 0 disables" contract to CacheSpec's ttl == 0 + // (disabled) rather than passing a negative value through (which CacheSpec reads as "no expiration"). + CacheSpec spec = ttlSeconds > 0 + ? CacheSpec.of(true, ttlSeconds, maxSize) + : CacheSpec.of(true, CacheSpec.CACHE_TTL_DISABLE_CACHE, maxSize); + this.entry = new MetaCacheEntry<>("iceberg-format", null, spec, + ForkJoinPool.commonPool(), false, true, 0L, true); + } + + /** Caching is on only when the TTL is positive; ttl-second <= 0 means "always infer live". */ + boolean isEnabled() { + return entry.stats().isEffectiveEnabled(); + } + + /** + * Returns the cached format name for {@code key} if present and unexpired, else runs {@code loader} (the live + * whole-table format inference), caches and returns it. Disabled cache -> {@code loader} every call. The + * loader runs OUTSIDE Caffeine's compute lock (single-flight per key) and its exception propagates unwrapped + * and is NOT cached. + */ + String getOrLoad(Key key, Supplier loader) { + return entry.get(key, ignored -> loader.get()); + } + + /** Drops every cached snapshot entry for one table so the next read infers live (REFRESH TABLE). */ + void invalidate(TableIdentifier id) { + entry.invalidateIf(key -> key.id.equals(id)); + } + + /** Drops every cached entry for one database (REFRESH DATABASE / DROP DATABASE); db match = namespace equality. */ + void invalidateDb(String dbName) { + Namespace ns = Namespace.of(dbName); + entry.invalidateIf(key -> key.id.namespace().equals(ns)); + } + + /** Drops all cached entries (REFRESH CATALOG). */ + void invalidateAll() { + entry.invalidateAll(); + } + + /** Test-only: current number of cached entries (accurate map membership, not Caffeine's estimate). */ + int size() { + int[] count = {0}; + entry.forEach((key, value) -> count[0]++); + return count[0]; + } + + /** Test-only: how many times the live loader (the whole-table format inference) actually ran — the metric gate. */ + long loadCountForTest() { + return entry.stats().getLoadSuccessCount(); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index 58a6848a0c3ad3..093f8a518fdf40 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -228,6 +228,10 @@ public class IcebergScanPlanProvider implements ConnectorScanPlanProvider { // when the connector's credential gate disables the cross-query layer; when null resolveTable still uses the // query-scoped fat handle and falls back to a direct remote load. private final IcebergTableCache tableCache; + // PERF-03: cross-query inferred-file-format cache shared with the connector, owned by the long-lived + // IcebergConnector and injected via getScanPlanProvider. Nullable — null via the offline-test ctors; when null + // getScanNodeProperties resolves file_format_type live (matching pre-PERF-03 behaviour, node-memoized per query). + private final IcebergFormatCache formatCache; // FIX-SCAN-METRICS: per-query stash of the iceberg SDK scan diagnostics captured by the attached // IcebergScanProfileReporter during planScan, keyed by session queryId. fe-core drains it @@ -280,12 +284,26 @@ public IcebergScanPlanProvider(Map properties, Function catalogOpsResolver, ConnectorContext context, IcebergManifestCache manifestCache, IcebergRewritableDeleteStash rewritableDeleteStash, IcebergTableCache tableCache) { + this(properties, catalogOpsResolver, context, manifestCache, rewritableDeleteStash, tableCache, null); + } + + /** + * Full ctor used by {@link IcebergConnector#getScanPlanProvider()}, adding the PERF-03 cross-query + * inferred-file-format cache ({@code formatCache}). The 6-arg ctor delegates here with a null format cache + * (offline tests + pre-cache paths resolve {@code file_format_type} live). + */ + public IcebergScanPlanProvider(Map properties, + Function catalogOpsResolver, + ConnectorContext context, IcebergManifestCache manifestCache, + IcebergRewritableDeleteStash rewritableDeleteStash, IcebergTableCache tableCache, + IcebergFormatCache formatCache) { this.properties = properties; this.catalogOpsResolver = catalogOpsResolver; this.context = context; this.manifestCache = manifestCache; this.rewritableDeleteStash = rewritableDeleteStash; this.tableCache = tableCache; + this.formatCache = formatCache; } /** @@ -1400,8 +1418,14 @@ public Map getScanNodeProperties( // V1-only reader bug. Emit the table's real data format (legacy IcebergScanNode.getFileFormatType // parity, same IcebergUtils.getFileFormat resolution, which throws on a non-parquet/orc table); the // per-file format still travels per range, since one table may mix parquet and orc data files. + // PERF-03: the non-system format resolution falls back to an unfiltered whole-table planFiles() when the + // table sets neither write-format nor write.format.default; memoize that inference per (table, snapshot) + // across queries via formatCache (pure metadata, no credential gate). Null cache (offline) resolves live. props.put("file_format_type", - systemTable ? "jni" : IcebergWriterHelper.getFileFormat(table).name().toLowerCase(Locale.ROOT)); + systemTable ? "jni" + : IcebergWriterHelper.getFileFormat(table, + TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), formatCache) + .name().toLowerCase(Locale.ROOT)); // [D-065] System (metadata) tables ($snapshots/$files/...) read via the JNI serialized-split path // (planSystemTableScan): the metadata-table schema travels INSIDE the serialized FileScanTask, so BE // needs neither the base-table path_partition_keys (a metadata table is not base-spec partitioned -> diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java index 3ac890e7282650..a069a29921b610 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java @@ -34,9 +34,11 @@ import org.apache.iceberg.PartitionData; import org.apache.iceberg.PartitionSpec; import org.apache.iceberg.Schema; +import org.apache.iceberg.Snapshot; import org.apache.iceberg.SortOrder; import org.apache.iceberg.Table; import org.apache.iceberg.TableProperties; +import org.apache.iceberg.catalog.TableIdentifier; import org.apache.iceberg.io.CloseableIterable; import org.apache.iceberg.io.WriteResult; import org.apache.iceberg.types.Type; @@ -45,6 +47,8 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.io.IOException; +import java.io.UncheckedIOException; import java.nio.ByteBuffer; import java.time.ZoneId; import java.util.ArrayList; @@ -263,8 +267,21 @@ static List convertToDeleteFiles(FileFormat format, PartitionSpec sp * from the current snapshot's data files (defaulting to parquet). Throws on a non-orc/parquet format. */ static FileFormat getFileFormat(Table table) { - Map properties = table.properties(); - String fileFormatName = resolveFileFormatName(table, properties); + return toFileFormat(resolveFileFormatName(table, table.properties())); + } + + /** + * PERF-03 cache-aware overload used by the scan path ({@code IcebergScanPlanProvider.getScanNodeProperties}). + * Identical resolution to {@link #getFileFormat(Table)}, except the whole-table {@code planFiles()} inference + * fallback (fired only when the table sets neither {@code write-format} nor {@code write.format.default}) is + * memoized per {@code (id, currentSnapshotId)} through {@code cache}. The two cheap property probes stay + * uncached. A {@code null} cache/id (offline tests) or an empty table (no current snapshot) resolves live. + */ + static FileFormat getFileFormat(Table table, TableIdentifier id, IcebergFormatCache cache) { + return toFileFormat(resolveFileFormatName(table, table.properties(), id, cache)); + } + + private static FileFormat toFileFormat(String fileFormatName) { if (fileFormatName.toLowerCase().contains(ORC_NAME)) { return FileFormat.ORC; } else if (fileFormatName.toLowerCase().contains(PARQUET_NAME)) { @@ -284,20 +301,70 @@ private static String resolveFileFormatName(Table table, Map pro return inferFileFormatFromDataFiles(table); } + /** + * PERF-03: like {@link #resolveFileFormatName(Table, Map)} but routes the inference fallback (the heavy + * unfiltered {@code planFiles()}) through the cross-query {@code (id, snapshotId)} cache. Only the inference is + * cached; the property probes are cheap. A failed inference (the loader throws) is NOT cached, so the next + * query retries — for this query it degrades to the parquet default, mirroring the swallow of the live path. + */ + private static String resolveFileFormatName(Table table, Map properties, + TableIdentifier id, IcebergFormatCache cache) { + if (properties.containsKey(WRITE_FORMAT)) { + return properties.get(WRITE_FORMAT); + } + if (properties.containsKey(TableProperties.DEFAULT_FILE_FORMAT)) { + return properties.get(TableProperties.DEFAULT_FILE_FORMAT); + } + Snapshot snapshot = table.currentSnapshot(); + if (cache == null || id == null || snapshot == null) { + return inferFileFormatFromDataFiles(table); + } + try { + return cache.getOrLoad(new IcebergFormatCache.Key(id, snapshot.snapshotId()), + () -> inferFirstDataFileFormat(table)); + } catch (RuntimeException e) { + LOG.warn("Failed to infer file format from data files for table {}, defaulting to {}", + table.name(), PARQUET_NAME, e); + return PARQUET_NAME; + } + } + + /** + * The whole-table format inference (the #64134 heavy op): reads the FIRST data file's format from the current + * snapshot via an unfiltered {@code planFiles()}. The swallowing wrapper used by the write path and the direct + * live path (a failure degrades to parquet). The cache loader instead calls {@link #inferFirstDataFileFormat} + * so a transient failure is not memoized. + */ private static String inferFileFormatFromDataFiles(Table table) { if (table.currentSnapshot() == null) { return PARQUET_NAME; } + try { + return inferFirstDataFileFormat(table); + } catch (RuntimeException e) { + LOG.warn("Failed to infer file format from data files for table {}, defaulting to {}", + table.name(), PARQUET_NAME, e); + return PARQUET_NAME; + } + } + + /** + * Raw first-data-file format inference that PROPAGATES failures (unchecked) so the cache loader does not + * memoize a transient remote-IO error. The caller guarantees a non-null current snapshot; an empty snapshot + * (no data files) yields the deterministic parquet default (cacheable). The try-with-resources + * {@code CloseableIterable.close()} checked {@link IOException} is wrapped as {@link UncheckedIOException} so + * the method stays unchecked (a {@code Supplier} loader cannot declare checked throws). + */ + private static String inferFirstDataFileFormat(Table table) { try (CloseableIterable files = table.newScan().planFiles()) { Iterator it = files.iterator(); if (it.hasNext()) { return it.next().file().format().name().toLowerCase(); } - } catch (Exception e) { - LOG.warn("Failed to infer file format from data files for table {}, defaulting to {}", - table.name(), PARQUET_NAME, e); + return PARQUET_NAME; + } catch (IOException e) { + throw new UncheckedIOException(e); } - return PARQUET_NAME; } /** diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java index 2b833db7c92149..61465b43abd3c3 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java @@ -292,4 +292,56 @@ public void refreshHooksInvalidatePartitionCache() { connector.invalidateAll(); Assertions.assertEquals(0, cache.size(), "REFRESH CATALOG drops everything"); } + + private static IcebergFormatCache.Key fmtKey(String db, String tbl, long snapshotId) { + return new IcebergFormatCache.Key(TableIdentifier.of(db, tbl), snapshotId); + } + + @Test + public void formatCacheBuiltForAllCatalogsIncludingCredentialGatedOnes() { + // The inferred-format cache stores a pure metadata format-name string (no FileIO/credential), so like the + // partition cache it is built for EVERY catalog -- including per-user session and REST vended-credentials, + // which disable the table cache. MUTATION: gating the format cache on the credential flags -> null -> red. + Assertions.assertNotNull( + new IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext()).formatCacheForTest(), + "a plain catalog builds the format cache"); + Map vended = new HashMap<>(); + vended.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST); + vended.put(IcebergConnectorProperties.REST_VENDED_CREDENTIALS_ENABLED, "true"); + Assertions.assertNotNull( + new IcebergConnector(vended, new RecordingConnectorContext()).formatCacheForTest(), + "a vended-credentials catalog still builds the format cache (a format name carries no credentials)"); + Map session = new HashMap<>(); + session.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST); + session.put(IcebergConnectorProperties.REST_SESSION, IcebergConnectorProperties.SESSION_USER); + Assertions.assertNotNull( + new IcebergConnector(session, new RecordingConnectorContext()).formatCacheForTest(), + "a per-user session catalog still builds the format cache"); + } + + @Test + public void refreshHooksInvalidateFormatCache() { + // The REFRESH hooks must clear the inferred-format cache too (else a rewrite that changed the write format + // would stay invisible beyond the pin): REFRESH TABLE drops that table's snapshot entries, REFRESH DATABASE + // that db's, REFRESH CATALOG everything. MUTATION: an invalidate* hook not touching formatCache -> a stale + // entry survives -> a size assert below red. + IcebergConnector connector = + new IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext()); + IcebergFormatCache cache = connector.formatCacheForTest(); + Assertions.assertNotNull(cache); + cache.getOrLoad(fmtKey("db1", "t1", 1L), () -> "parquet"); + cache.getOrLoad(fmtKey("db1", "t1", 2L), () -> "orc"); + cache.getOrLoad(fmtKey("db1", "t2", 1L), () -> "parquet"); + cache.getOrLoad(fmtKey("db2", "t1", 1L), () -> "parquet"); + Assertions.assertEquals(4, cache.size()); + + connector.invalidateTable("db1", "t1"); + Assertions.assertEquals(2, cache.size(), "REFRESH TABLE drops both snapshot entries of db1.t1"); + + connector.invalidateDb("db1"); + Assertions.assertEquals(1, cache.size(), "REFRESH DATABASE drops db1's remaining table"); + + connector.invalidateAll(); + Assertions.assertEquals(0, cache.size(), "REFRESH CATALOG drops everything"); + } } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergFormatCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergFormatCacheTest.java new file mode 100644 index 00000000000000..be8483a44e11c7 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergFormatCacheTest.java @@ -0,0 +1,181 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.iceberg.catalog.TableIdentifier; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Unit tests for {@link IcebergFormatCache} (PERF-03). Mirrors {@link IcebergPartitionCacheTest} but keys by + * {@code (TableIdentifier, snapshotId)} and stores the inferred format-name String. Covers within-TTL stability, + * the {@code ttl <= 0} disable, invalidation, and the not-cached-on-failure guarantee that makes a transient + * remote-IO failure retry on the next query (legacy parity). + */ +public class IcebergFormatCacheTest { + + private static IcebergFormatCache.Key key(String db, String tbl, long snapshotId) { + return new IcebergFormatCache.Key(TableIdentifier.of(db, tbl), snapshotId); + } + + @Test + public void cachesWithinTtlAndServesTheSameValue() { + AtomicInteger loads = new AtomicInteger(); + IcebergFormatCache c = new IcebergFormatCache(100, 1000); + + String first = c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return "orc"; + }); + // Second read of the SAME (table, snapshot) within TTL returns the cached format, NOT a fresh whole-table + // planFiles() inference -> this is what collapses the per-query #64134 fallback. MUTATION: inferring live + // every call -> returns "parquet" / loads==2 -> red. + String second = c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return "parquet"; + }); + Assertions.assertEquals("orc", first); + Assertions.assertEquals("orc", second, "within TTL the cached format must be served"); + Assertions.assertEquals(1, loads.get(), "the live inference must run exactly once within TTL"); + Assertions.assertEquals(1, c.loadCountForTest(), "the metric-gate load count must be 1"); + Assertions.assertTrue(c.isEnabled()); + } + + @Test + public void differentSnapshotIdIsADifferentKey() { + AtomicInteger loads = new AtomicInteger(); + IcebergFormatCache c = new IcebergFormatCache(100, 1000); + c.getOrLoad(key("db", "t", 1L), () -> { + loads.incrementAndGet(); + return "parquet"; + }); + // A new commit yields a new snapshot id -> a distinct key -> a live inference (freshness across snapshots, + // e.g. a rewrite that changed the write format). MUTATION: keying by table only -> loads==1 -> red. + String s2 = c.getOrLoad(key("db", "t", 2L), () -> { + loads.incrementAndGet(); + return "orc"; + }); + Assertions.assertEquals("orc", s2); + Assertions.assertEquals(2, loads.get()); + Assertions.assertEquals(2, c.size()); + } + + @Test + public void ttlZeroDisablesCachingAlwaysLive() { + AtomicInteger loads = new AtomicInteger(); + IcebergFormatCache c = new IcebergFormatCache(0, 1000); + c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return "orc"; + }); + String second = c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return "parquet"; + }); + // ttl-second=0 (the no-cache catalog) infers live every time. MUTATION: caching despite ttl<=0 -> + // second=="orc" / loads==1 -> red. + Assertions.assertEquals("parquet", second, "ttl-second=0 must always infer live"); + Assertions.assertEquals(2, loads.get()); + Assertions.assertFalse(c.isEnabled()); + Assertions.assertEquals(0, c.size(), "ttl-second=0 must not store anything"); + } + + @Test + public void negativeTtlDisablesCachingAlwaysLive() { + AtomicInteger loads = new AtomicInteger(); + IcebergFormatCache c = new IcebergFormatCache(-1, 1000); + c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return "orc"; + }); + String second = c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return "parquet"; + }); + Assertions.assertEquals("parquet", second, "ttl-second=-1 must always infer live"); + Assertions.assertEquals(2, loads.get()); + Assertions.assertFalse(c.isEnabled()); + } + + @Test + public void invalidateDropsAllSnapshotsOfOneTable() { + AtomicInteger loads = new AtomicInteger(); + IcebergFormatCache c = new IcebergFormatCache(100, 1000); + c.getOrLoad(key("db", "t", 1L), () -> "parquet"); + c.getOrLoad(key("db", "t", 2L), () -> "orc"); + c.getOrLoad(key("db", "other", 1L), () -> "parquet"); + Assertions.assertEquals(3, c.size()); + + // REFRESH TABLE db.t must drop BOTH snapshot entries of db.t and leave db.other intact. MUTATION: + // invalidating a single (table, snapshot) key -> the other snapshot survives -> size 2 here -> red. + c.invalidate(TableIdentifier.of("db", "t")); + Assertions.assertEquals(1, c.size(), "only db.other's entry must survive"); + String reload = c.getOrLoad(key("db", "t", 1L), () -> { + loads.incrementAndGet(); + return "orc"; + }); + Assertions.assertEquals("orc", reload); + Assertions.assertEquals(1, loads.get()); + } + + @Test + public void invalidateDbClearsOnlyThatDbsTables() { + IcebergFormatCache c = new IcebergFormatCache(100, 1000); + c.getOrLoad(key("db1", "t1", 1L), () -> "parquet"); + c.getOrLoad(key("db1", "t2", 1L), () -> "orc"); + c.getOrLoad(key("db2", "t1", 1L), () -> "parquet"); + Assertions.assertEquals(3, c.size()); + c.invalidateDb("db1"); + Assertions.assertEquals(1, c.size(), "only db2's entry must survive REFRESH DATABASE db1"); + } + + @Test + public void invalidateAllClearsEverything() { + IcebergFormatCache c = new IcebergFormatCache(100, 1000); + c.getOrLoad(key("db", "t1", 1L), () -> "parquet"); + c.getOrLoad(key("db", "t2", 1L), () -> "orc"); + Assertions.assertEquals(2, c.size()); + c.invalidateAll(); + Assertions.assertEquals(0, c.size()); + } + + @Test + public void loaderFailureIsNotCachedSoNextQueryRetries() { + // A transient remote-IO failure during inference (planFiles/close) must NOT be memoized, or the parquet + // fallback would stick for the whole TTL. The MetaCacheEntry manual-miss-load path re-throws the loader's + // RuntimeException verbatim and does not store it. MUTATION: caching on failure -> the second call would + // not re-run the loader / size==1 -> red. + IcebergFormatCache c = new IcebergFormatCache(100, 1000); + AtomicInteger loads = new AtomicInteger(); + Assertions.assertThrows(RuntimeException.class, () -> c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + throw new RuntimeException("transient manifest read failure"); + })); + Assertions.assertEquals(0, c.size(), "a failed inference must not be cached"); + // The next query at the same key re-runs the loader (retry), and a success is then cached. + String ok = c.getOrLoad(key("db", "t", 5L), () -> { + loads.incrementAndGet(); + return "orc"; + }); + Assertions.assertEquals("orc", ok); + Assertions.assertEquals(2, loads.get(), "the loader must re-run after a failure (no sticky failure)"); + Assertions.assertEquals(1, c.size()); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWriterHelperTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWriterHelperTest.java index dd6fde1b974db8..2fedacfb299ee0 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWriterHelperTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWriterHelperTest.java @@ -23,6 +23,7 @@ import com.google.common.base.VerifyException; import org.apache.iceberg.DataFile; +import org.apache.iceberg.DataFiles; import org.apache.iceberg.DeleteFile; import org.apache.iceberg.FileFormat; import org.apache.iceberg.PartitionSpec; @@ -245,4 +246,84 @@ public void getFileFormatThrowsOnUnsupported() { Assertions.assertThrows(RuntimeException.class, () -> IcebergWriterHelper.getFileFormat(tableWith("write.format.default", "avro"))); } + + // ─────────────────── getFileFormat: PERF-03 cross-query inference cache ─────────────────── + + /** Unpartitioned table carrying one data file of {@code fileFormat} and the given extra properties. */ + private Table tableWithDataFile(FileFormat fileFormat, String... props) { + Table table = tableWith(props); + table.newAppend().appendFile(DataFiles.builder(unpartitionedSpec) + .withPath("s3://b/db1/t/f0." + fileFormat.name().toLowerCase()) + .withFileSizeInBytes(100) + .withRecordCount(1) + .withFormat(fileFormat) + .build()).commit(); + return table; + } + + @Test + public void getFileFormatInferenceIsCachedAcrossQueriesAtSameSnapshot() { + // A table with NO write-format / write.format.default (migrated / engine-default table) resolves the scan + // level file_format_type via an unfiltered whole-table planFiles() inference (the #64134 heavy op). PERF-03 + // memoizes that inference per (table, currentSnapshotId): repeated getScanNodeProperties computes across + // queries collapse onto ONE remote inference. MUTATION: not threading the cache into the call -> each query + // re-scans -> loadCountForTest > 1 -> red. + Table table = tableWithDataFile(FileFormat.ORC); + TableIdentifier id = TableIdentifier.of("db1", "t"); + IcebergFormatCache cache = new IcebergFormatCache(100, 1000); + + FileFormat f1 = IcebergWriterHelper.getFileFormat(table, id, cache); + FileFormat f2 = IcebergWriterHelper.getFileFormat(table, id, cache); + FileFormat f3 = IcebergWriterHelper.getFileFormat(table, id, cache); + + Assertions.assertEquals(FileFormat.ORC, f1); + Assertions.assertEquals(FileFormat.ORC, f2); + Assertions.assertEquals(FileFormat.ORC, f3); + Assertions.assertEquals(1, cache.loadCountForTest(), + "the whole-table inference must run exactly once across repeated queries at one snapshot"); + Assertions.assertEquals(1, cache.size(), "one (table, snapshot) entry"); + // Parity: the cached resolution matches the uncached (live) resolution. + Assertions.assertEquals(IcebergWriterHelper.getFileFormat(table), f1, + "cached format must equal a live (uncached) inference"); + } + + @Test + public void getFileFormatPropertyPathIsNeverCached() { + // When the table carries a format property, resolution is a cheap property read that must NOT touch the + // cache (only the inference fallback is memoized). MUTATION: caching the property path -> size > 0 -> red. + Table table = tableWithDataFile(FileFormat.ORC, "write.format.default", "parquet"); + TableIdentifier id = TableIdentifier.of("db1", "t"); + IcebergFormatCache cache = new IcebergFormatCache(100, 1000); + + Assertions.assertEquals(FileFormat.PARQUET, IcebergWriterHelper.getFileFormat(table, id, cache)); + Assertions.assertEquals(0, cache.size(), "the property path must never populate the inference cache"); + Assertions.assertEquals(0, cache.loadCountForTest()); + } + + @Test + public void getFileFormatNullCacheResolvesLive() { + // Offline / pre-cache paths pass a null cache: resolution stays live and correct (no NPE). + Table table = tableWithDataFile(FileFormat.ORC); + Assertions.assertEquals(FileFormat.ORC, + IcebergWriterHelper.getFileFormat(table, TableIdentifier.of("db1", "t"), null)); + } + + @Test + public void getFileFormatUnsupportedFirstFileCachesInferenceButRethrowsEachCall() { + // R3: an unsupported first data-file format (e.g. avro) is a SUCCESSFUL inference (the loader returns + // "avro"); the unsupported-format throw happens in the format mapping OUTSIDE getOrLoad. So the inference is + // cached once (loadCount == 1) yet every call re-throws (parity with legacy, which inferred + threw every + // query) without re-scanning. MUTATION: caching the mapped value / caching the throw -> loadCount != 1. + Table table = tableWithDataFile(FileFormat.AVRO); + TableIdentifier id = TableIdentifier.of("db1", "t"); + IcebergFormatCache cache = new IcebergFormatCache(100, 1000); + + Assertions.assertThrows(RuntimeException.class, + () -> IcebergWriterHelper.getFileFormat(table, id, cache)); + Assertions.assertThrows(RuntimeException.class, + () -> IcebergWriterHelper.getFileFormat(table, id, cache)); + Assertions.assertEquals(1, cache.loadCountForTest(), + "the inference runs once; the unsupported-format throw is in the mapping, not the loader"); + Assertions.assertEquals(1, cache.size()); + } } From 586ad5f156d18100671b5a1221d44eb702215d31 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 07:25:55 +0800 Subject: [PATCH 09/47] [doc](catalog) fe-connector-iceberg PERF-03: design + summary + tracker updates; next = PERF-04 Records the file-format inference cache work (PERF-03, commit 0b96f2e6c78): design doc (with the red-team fold-ins R1/R2/R3), summary, tasklist check-off, progress log, and a fresh HANDOFF pointing at PERF-04 (the two IcebergManifestCache bypasses, C17/C18) with the OOM-risk open question flagged. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 50 ++++--- ...X-PERF-03-format-inference-cache-design.md | 127 ++++++++++++++++++ ...-PERF-03-format-inference-cache-summary.md | 58 ++++++++ plan-doc/perf-hotpath-iceberg/progress.md | 13 ++ plan-doc/perf-hotpath-iceberg/tasklist.md | 11 +- 5 files changed, 232 insertions(+), 27 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-03-format-inference-cache-design.md create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-03-format-inference-cache-summary.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index 2b1b91f2278046..aabe24ba15d4c5 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -6,46 +6,52 @@ --- -# ✅ 已完成（session 2，2026-07-18） +# ✅ 已完成（session 1~3） -- **PERF-01**（commit `484f0e0c125`）：胖 handle（`IcebergTableHandle.transient resolvedTable`，查询内）+ 跨查询 `IcebergTableCache`（挂 `IcebergConnector`，凭证 gate）。同表规划期远端 `loadTable` 3~7→1。 -- **PERF-02**（commit `518d0599cbf`）：跨查询 `IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、值 raw 分区列表、**无凭证 gate**——纯元数据）。分区表分析期 PARTITIONS 扫描每查询→每快照一遍；MTMV 一次 refresh 的 4~6 遍→1。小结见 `designs/FIX-PERF-02-*-summary.md`。 -- 两轮均全 iceberg 模块单测绿（943/0/1），checkstyle 绿，0 回归。 -- **可复用产物**：`IcebergTableCache`(键 TableIdentifier) / `IcebergPartitionCache`(键 `(TableIdentifier,snapshotId)`+`loadCountForTest` 度量守门) 两套 `MetaCacheEntry` 基建；凭证 gate 判据（值含 FileIO→gate，纯元数据→无 gate）；胖 handle 携带模式。PERF-03/10 复用。 +- **PERF-01**（commit `484f0e0c125`）：胖 handle（`IcebergTableHandle.transient resolvedTable`，查询内）+ 跨查询 `IcebergTableCache`（挂 `IcebergConnector`，**凭证 gate**）。同表规划期远端 `loadTable` 3~7→1。 +- **PERF-02**（commit `518d0599cbf`）：跨查询 `IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、值 raw 分区列表、**无凭证 gate**）。分区表分析期 PARTITIONS 扫描每查询→每快照一遍；MTMV 4~6 遍→1。 +- **PERF-03**（commit `0b96f2e6c78`）：跨查询 `IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、值格式名 `String`、**无凭证 gate**）。无 `write-format`/`write.format.default` 的表每查询整表格式推断 `planFiles()`→每快照一遍。只缓存推断兜底、失败不缓存、映射/抛点在 `getOrLoad` 之外；写路径 7 调用点未动（留 PERF-07）。小结 `designs/FIX-PERF-03-*-summary.md`。 +- 三轮均全 iceberg 模块单测绿（932→943→957 / 0 fail / 1 skip），checkstyle 绿，0 回归。 +- **可复用产物**：三套 `(TableIdentifier[,snapshotId])` + `MetaCacheEntry` 缓存基建（table/partition/format）；**凭证 gate 判据**（值含 FileIO/凭证→gate；纯元数据→无 gate，已三次印证）；「缓存只存 loader 原始输出、映射与抛点放 getOrLoad 之外、失败透传 unchecked 不入缓存」范式；`loadCountForTest` 度量守门法。 --- -# 🆕 下一个 session = **PERF-03（#64134 planFiles 兜底复活，C2 C11）** +# 🆕 下一个 session = **PERF-04（IcebergManifestCache 两条旁路接回，C17 C18）** + +## ⚠️ 触发面很窄（P1）：仅当用户显式开 `meta.cache.iceberg.manifest.enable=true`（**默认 OFF**）。修的是"缓存开了却在两条路径上被绕过"。 ## 第一件事（立项流程见 README §单项立项流程） -1. **复核（动码前，行号信 grep 不信文档）**：按 findings.json C2/C11 重新 grep：`getScanNodeProperties → IcebergWriterHelper.getFileFormat → resolveFileFormatName → inferFileFormatFromDataFiles → table.newScan().planFiles()`（无过滤**整表 manifest 扫描**，只为拿"第一个数据文件的格式"）。门槛 = 表属性无 `write-format` 且无 `write.format.default`（迁移表 + 任何写引擎从不显式设该属性的表）。乘数 = 每次 properties 计算 1 次（无谓词 1/查询、带谓词 2/查询）。确认成本/乘数/纯冗余（planScan 自己的 planFiles 枚举里每个 `dataFile.format()` 都有）。 -2. **⚠ 依赖已过时**：tasklist 写"依赖 PERF-01 的 convertPredicate 收窄消第二次"——但 **PERF-01 已删 Part B（convertPredicate 收窄，红队证伪为 no-op）**，故不存在"第二次"可消；PERF-03 就是把这一次整表扫描本身干掉。 -3. **设计**（写 `designs/FIX-PERF-03-...-design.md`）：两条路线择一或结合——(a) 按 `(table UUID/TableIdentifier, snapshotId)` memoize 解析出的 format（快照不变⇒格式不变，复用 PERF-02 的 `(table,snapshotId)` 键+cache 套路，值=一个 `FileFormat`/字符串，纯元数据无 gate）；或 **(b) 从 planScan 自己的 split 枚举结果反推 scan 级 format 传下去**（更彻底：连那"一次"整表扫描都不做，因为数据规划本就要枚举 dataFile）。**倾向 (b)**（消除而非缓存重操作，更符合问题类），但要核 `getScanNodeProperties`(属性路径) 与 `planScan`(数据路径) 的先后/是否同 handle 血缘能共享枚举结果——若两路径独立触发则 (b) 需跨路径传递，退回 (a) memoize。 -4. **红队 + TDD**：度量守门 = 无 write-format 的表规划期 `planFiles`（或 format 解析）远端次数从 1→0（(b)）或跨查询 1→命中（(a)）。可仿 PERF-02 的 `loadCountForTest` 计数法。 -5. 守铁律：memo/派生放连接器侧；fe-core 不解析属性（format 组装在插件侧→thrift）。 +1. **复核（动码前，行号信 grep 不信文档；PERF-01/02/03 后行号已漂移）**。两条独立旁路，同一病根（manifest cache 只接在同步 `planFileScanTask` 分支，`gate isManifestCacheEnabled`）： + - **C17（streaming 大表踢出 cache）**：`FileQueryScanNode.createScanRangeLocations isBatchMode → PluginDrivenScanNode.computeBatchMode（streamingSplitEstimate ≥ num_files_in_batch_mode → streamingBatch，无 cache gate）→ startStreamingSplit → IcebergScanPlanProvider.streamSplits → scan.planFiles()`（裸 planFiles，**不走 cache**）。**讽刺点**：文件数 ≥ `num_files_in_batch_mode`（默认 1024）才走 streaming，而这**恰好是 manifest cache 想加速的大表**；legacy 在 batch 模式仍走 cache。 + - **C18（COUNT(*) 下推绕过 cache）**：`getSplits planScan(countPushdown=true) → planScanInternal（getCountFromSnapshot≥0）→ planCountPushdown → scan.planFiles()`，在到达 cache 分支（`planFileScanTask`）**之前 return**；且 `ParallelIterable` 激进提交所有 manifest 读任务（虽只要计数聚合）。 +2. **设计**（写 `designs/FIX-PERF-04-...-design.md`）。审计建议方向： + - C17 = **cache 开启时 `streamingSplitEstimate` 返回 -1**（= 本模块"count 不可下推/未知"同款 sentinel），令 `computeBatchMode` 退回同步物化 `planFileScanTask` 路径（走 cache）。**⚠ 关键风险须论证**：streaming 是**大表 OOM 保护**（不持全量 task 列表）。强制大表走同步物化会否重新引入 OOM？—— 必须核 legacy「batch 模式如何同时用 cache 又不 OOM」（读 `git show 83585fd5097^:.../IcebergScanNode` 附近），确认"cache 开→退同步"是否真 legacy-parity、还是要更细（如仅当 cache 命中才退）。这条别当"一行修复"轻下。 + - C18 = count 分支改走 `planFileScanTask`（honors cache），而非裸 `scan.planFiles()`。较独立、风险小，可先做。 +3. **红队 + TDD**：度量守门 = manifest cache 开启时，streaming 大表 / COUNT(*) 下推的 manifest 读**命中 cache**（非每次裸 planFiles）。可仿现有 `IcebergManifestCache` 测试 + `getManifestCacheValue` 计数。 +4. 守铁律：改动放**连接器侧**（`IcebergScanPlanProvider` 的 `streamingSplitEstimate`/`planCountPushdown`）；C17 的 -1 sentinel 是连接器决策，别在 fe-core 加 source-specific 分支。 -## ⚠️ 关键认知（承 PERF-01/02） +## ⚠️ 关键认知（承 PERF-01/02/03） -- **缓存都挂 `IcebergConnector`**（长生命周期）；键统一用 `(TableIdentifier[, snapshotId])`，snapshotId 由 `IcebergLatestSnapshotCache` 稳定。别靠 handle 跨查询（handle 每查询新建）。 -- **凭证 gate 判据**：缓存值**含 FileIO/凭证**（如 raw Table）→ gate 掉 `session=user`/`vended`；**纯元数据**（快照 id、分区列表、format）→ 无 gate。PERF-03 的 format 是纯元数据 → 无 gate。 -- **测试禁共享可变 handle/静态态**：胖 handle 后 handle 带可变状态，跨用例复用的 `static` handle 会串味（PERF-01 踩过 `T1`）。 -- **scan 级 format_type 决定 V1/V2 scanner**（见全局 memory）：PERF-03 若走 (b) 反推 format 传下去，务必别把整个连接器错钉在 `file_format_type=jni`（只 isSystemTable 该发 jni），对齐上游同名 ScanNode 的 `getFileFormatType()`。 +- **缓存都挂 `IcebergConnector`**（长生命周期）；键统一 `(TableIdentifier[, snapshotId])`。`IcebergManifestCache` 是**已存在**的（path-keyed、no-TTL、cap 100k），PERF-04 不新建缓存，只是把两条绕过它的路径接回去。 +- **凭证 gate 判据**：值含 FileIO/凭证→gate；纯元数据→无 gate。（PERF-04 不涉新缓存值，无关。） +- **PERF-05（C9 information_schema comment 每表 loadTable）改动小、收益明确**，若 PERF-04 的 OOM 风险论证卡住，可临时插队 PERF-05 热身（README 允许按收益/风险微调顺序，但默认按 §5 优先级）。 +- **scan 级 format_type 决定 V1/V2 scanner**（全局 memory）：PERF-04 改 streaming/count 决策**不碰** format_type，但注意 streaming↔batch 模式切换别误动 `getFileFormatType` 路径。 --- # 🧰 构建/验证坑（**实证，务必照做**） -1. **可靠跑单测 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`**（绝对 `-f /fe/pom.xml`）。原因：本 worktree `${revision}` CI 版本 + 未 flatten 的已装 pom → `-pl iceberg` 单模块解析不到 `fe-connector:pom:${revision}`（`-Drevision=` 不透传传递依赖），必须 `-am`；且 `-am test` 只到 test 相不产 hms-hive-shade 的 shade jar（缺 `HiveConf`），故用 **`install`**（到 package 相跑 shade）。`-Dtest=`（=`ls src/test/.../*Test.java`）让上游 0 匹配测试快跳。**checkstyle 单独跑** `mvn checkstyle:check -pl ... -am`（测试跑时 `-Dcheckstyle.skip=true`）。 +1. **可靠跑单测 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`**（绝对 `-f /fe/pom.xml`）。原因：本 worktree `${revision}` CI 版本 + 未 flatten 的已装 pom → `-pl iceberg` 单模块解析不到 `fe-connector:pom:${revision}`，必须 `-am`；且 `-am test` 只到 test 相不产 hms-hive-shade 的 shade jar（缺 `HiveConf`），故用 **`install`**。`-Dtest=`（=`ls src/test/.../*Test.java` 拼逗号，本轮 55 类）让上游 0 匹配测试快跳。**checkstyle 单独跑** `mvn checkstyle:check -pl fe-connector/fe-connector-iceberg`（测试跑时 `-Dcheckstyle.skip=true`）。 2. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 静默跳过）。 -3. **后台 task 通知的 "exit code" 是尾部命令的**，非 maven —— 读日志 `BUILD SUCCESS`/`Tests run:` 行（`Monitor` 用 `until grep -qE 'BUILD SUCCESS|BUILD FAILURE'` 守 5s 轮询）。 -4. **checkstyle**：LineLength(120) 对 **test 源已 suppress**；主源 ≤120；import 序 `SAME_PACKAGE(3)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行（`org.apache.doris.*`=SAME_PACKAGE，含嵌套类 import 不算 redundant）。 -5. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。 -6. **并发探测**：`pgrep -a -f maven`。注意 `/mnt/disk1/yy/git/doris`（另一 worktree）常有构建在跑——那不动本 `wt-catalog-spi` 的文件，非冲突；只看**本目录** `find fe -newermt '-120 sec'`。 +3. **别用 `nohup ... &` 套在 Bash `run_in_background` 里**（本轮踩坑）：外层会立即随 `echo` 退出、maven 变孤儿仍在跑 → "task completed exit 0" 是假信号。直接 `mvn ... > log 2>&1`（run_in_background）或用 `Monitor`/`until grep -qE 'BUILD SUCCESS|BUILD FAILURE' log` 守。读日志 `BUILD SUCCESS`/`Tests run:` 行，别信 exit code。 +4. **checkstyle**：LineLength(120) 对 **test 源已 suppress**；主源 ≤120；import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA`，组内字母序、组间空行（`org.apache.iceberg.catalog.X` 排在 `org.apache.iceberg.T*` 之后、`org.apache.iceberg.io.*` 之前）。 +5. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add 改动文件。 +6. **并发探测**：本目录 `find fe -newermt '-120 sec'` + `pgrep -a -f maven`。`/mnt/disk1/yy/git/doris` 是另一 worktree 的构建，不动本 `wt-catalog-spi` 文件，非冲突。 7. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 --- # 🗂 开放问题 -- 无。PERF-01/02 已合，进入 PERF-03。 +- **PERF-04 C17 的 OOM 风险**（见上第一件事 step 2）：cache 开启时把大表从 streaming 退回同步物化，是否重新引入 OOM？须核 legacy batch 模式如何兼顾 cache 与不 OOM，再定"退同步"的精确条件。这是 PERF-04 设计的核心待答项。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-03-format-inference-cache-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-03-format-inference-cache-design.md new file mode 100644 index 00000000000000..03d18401c9c82c --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-03-format-inference-cache-design.md @@ -0,0 +1,127 @@ +# FIX-PERF-03 — Design（`file_format_type` 兜底整表 planFiles 跨查询 memoize） + +> 覆盖审计发现 **C2 / C11**（同一重操作的两个视角）。权威分析见审计报告簇2 + `findings.json` 的 +> `IcebergWriterHelper.getFileFormat planFiles() fallback` / `getScanNodeProperties infers file_format_type via +> whole-table planFiles()` 两条。基线 HEAD `cab6e1ed50e`（PERF-01/02 已合）。行号以本次 grep 为准。 + +## Problem + +每条对 iceberg 表的 `SELECT`/`EXPLAIN` 在 scan-node 初始化期都要产出 scan 级 `file_format_type` +（`IcebergScanPlanProvider.getScanNodeProperties:1329-1330`）。当表属性里**既无** `write-format` +昵称、**又无** `write.format.default`（= 迁移表 + 任何写引擎从不显式写该属性的表，parquet 是隐式默认不入元数据） +时，解析走 `IcebergWriterHelper.getFileFormat → resolveFileFormatName:284 → inferFileFormatFromDataFiles:287 +→ table.newScan().planFiles():291` —— **一次无过滤的整表 manifest 扫描**，只为读**第一个数据文件的格式**。 + +- **重**：`planFiles()` 读该快照的 manifest-list + manifest（经 FileIO 远端对象存储 IO）；iceberg 的 + `ParallelIterable` 还会把 manifest reader 激进 fan-out 到 worker 池，虽然只消费第一个 task 就 close。 + 大表（上千 manifest）每查询数百 ms~秒级远端 IO。 +- **乘数**：每次 `getScanNodeProperties` 计算 1 次。该结果在 fe-core 侧被 node 级 memoize + （`PluginDrivenScanNode.cachedPropertiesResult`），故**查询内 1 次**；但**跨查询零缓存** —— 每条查询/EXPLAIN 重扫一遍。 +- **纯冗余的两个来源**： + 1. `planScan` 自己的（带谓词的）`planFiles` 枚举里，每个 `FileScanTask` 的 `dataFile.format()` 都已带格式； + 2. 每个 range 在 `buildRange:1092` 已按 `dataFile.format()` 逐文件下发格式。 + scan 级这个值只用来选 BE 的 V1/V2 scanner（`file_format_type` gate，见全局 memory），per-file 格式仍逐 range 走。 + +这是 #64134（旧 `IcebergUtils.getFileFormat` 的 planFiles 兜底，DORIS-27138 同型）在新框架以 per-query 乘数复活。 + +## Root Cause + +新框架 SPI 无任何按 `(表, 快照)` 的格式 memo。快照不可变 ⇒ 该快照第一个数据文件的格式是**纯元数据、恒定**， +却被每条查询重新远端解析。legacy 侧此解析同样无缓存，但重构把它从"偶发"抬到了"每条 getScanNodeProperties 必经"。 + +## 为什么选"跨查询 memoize"（路线 a）而非"从 planScan 枚举反推"（路线 b） + +HANDOFF 原倾向路线 b（消除而非缓存），但**复核否决 b**，落到 a，理由是 fe-core scan 生命周期的**时序**： + +`FileQueryScanNode.createScanRangeLocations()` 里 **先** `getFileFormatType()`（**line 325** → 触发本重操作）， +**后** `getSplits()`（**line 422** → 触发 `planScan`）。即 scan 级格式在 `planScan` **之前**就必须给出。 + +- 路线 b 要"复用 planScan 的枚举结果"，就得让 `getFileFormatType` 等 `planScan` 先跑、或 `planScan` 先跑再回填 —— + 这需要**改 fe-core 时序或跨路径传值**，违反铁律「fe-core 源只出不进」，且是 source-specific reorder。 +- 且两处语义不同：格式推断是 `table.newScan()`（**无过滤**、`currentSnapshot`），`planScan` 是**带谓词**扫描。 + 谓词剪空所有文件时，从 `planScan` 反推会拿不到格式 → 与 legacy「无过滤兜底总能找到首文件格式」**破 parity**。 + +结论：正是 HANDOFF 预判的「两路径独立触发 → 退回 (a) memoize」。路线 a 无 fe-core 改动、无时序依赖、parity 严格。 + +## Design（连接器侧，零 fe-core 改动；复用 PERF-02 套路） + +### 1. 新增 `IcebergFormatCache`（挂 `IcebergConnector`） +- 键 `(TableIdentifier, snapshotId)`，**值 = `String`**（`inferFileFormatFromDataFiles` 的原始输出格式名）。 +- 结构逐字镜像 `IcebergPartitionCache`：`MetaCacheEntry`、TTL = `meta.cache.iceberg.table.ttl-second` + （`<=0` 关）、`isEnabled/getOrLoad/invalidate/invalidateDb/invalidateAll/size/loadCountForTest`。 +- **无凭证 gate**（同 PERF-02）：值是纯元数据（格式名字符串），不含 FileIO/凭证 → 对所有目录无条件构造，仅 TTL 控。 + +### 2. `IcebergWriterHelper` 加 cache-aware 重载（镜像 PERF-02 的 `loadRawPartitions` 拆包装） +- 保留现有 `getFileFormat(Table)`（写路径 + 离线继续用，行为字节不变）。 +- 新增 `getFileFormat(Table, TableIdentifier, IcebergFormatCache)`： + - `resolveFileFormatName` 里**属性优先**（`write-format` / `write.format.default` 两条 cheap 检查**不进缓存**）； + - **仅 inference 兜底**这条走缓存：键 `(id, table.currentSnapshot().snapshotId())`，loader = 抛异常版推断 + `inferFirstDataFileFormat(table)`（`currentSnapshot==null`/`cache==null`/`id==null` → 绕过缓存走原逻辑）。 +- **失败不入缓存**（严守 PERF-02 纪律）：把现有 `inferFileFormatFromDataFiles`（吞异常返回 parquet，写路径保留） + 重构成「`inferFirstDataFileFormat`（**透传异常**）+ 外层 try/catch 吞成 parquet 默认」；cache-aware 路径直接 + 用透传版做 loader，`MetaCacheEntry` 对抛异常的 loader **不缓存** → 瞬时 IO 失败下条查询会重试（legacy parity）。 + 空快照（有 snapshot 但零数据文件）→ 确定性 parquet，**可缓存**。 + - **R1（红队实现坑）**：loader 是 `Supplier`，只能抛 unchecked。`MetaCacheEntry.loadAndTrack` 只捕 + `RuntimeException|Error` 后**重抛且不 put**（`MetaCacheEntry:337-343`），故 `inferFirstDataFileFormat` 里 + try-with-resources 的 `CloseableIterable.close()` 抛的 **checked `IOException` 必须包成 `UncheckedIOException`** + 再抛（iceberg 的 `planFiles/iterator/next` 本就抛 unchecked，直接透传即可），否则既编译不过、又破"失败不缓存"。 + +### 3. `IcebergScanPlanProvider` +- 加 `private final IcebergFormatCache formatCache;`（nullable，离线 ctor 传 null，同 `tableCache`）。 +- 主 ctor（`IcebergConnector.getScanPlanProvider` 用的 7 参）加 `formatCache` 参；便利 ctor 链默认 null。 +- **唯一改的调用点** `getScanNodeProperties:1329-1330`：非系统表分支 + `IcebergWriterHelper.getFileFormat(table)` → `IcebergWriterHelper.getFileFormat(table, + TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), formatCache)`。系统表仍 `"jni"` 不变。 + +### 4. `IcebergConnector` +- field `formatCache`；构造 `new IcebergFormatCache(resolveTableCacheTtlSecond(properties), DEFAULT_TABLE_CACHE_CAPACITY)`（无 gate）。 +- `getScanPlanProvider` 传 `formatCache`；`invalidateTable/Db/All` 各加 `formatCache.invalidate*`；`formatCacheForTest()`。 + +### 写路径不动（范围边界） +`IcebergConnectorTransaction:658/690`、`IcebergWriterHelper:91`、`IcebergWritePlanProvider:400/464/525/746` +七个写路径 `getFileFormat(Table)` 调用点**保持无缓存重载**——那是 **PERF-07（C20，写路径 3~5 次 load）**的范围， +本任务只碰 scan 路径的那一个点。 + +## Parity 论证 + +- **同快照同结果**：键含 snapshotId，快照不可变；新提交→新 snapshotId→新键→实扫。缓存值 `String`，消费方 + `getFileFormat` 仍做 orc/parquet 映射（含对 `avro` 等不支持格式的 throw）——映射在 `getOrLoad` **之外**， + 故缓存永不存"会抛"的值、抛点行为与 legacy 完全一致。 +- **混格式表（parquet+orc 混存）**：`inferFileFormatFromDataFiles` 取"第一个 task"的格式，`ParallelIterable` + 多线程下"第一个"本就不确定 → legacy 该值对混格式表**本就是任意的**；缓存把它固定成每快照一个确定值。 + scan 级值只选 V1/V2，per-file 格式仍逐 range 走，混格式表**读取正确性不受影响**（correctness-中性）。 + - **R2（红队精确化）**：这是**可复现性(reproducibility)的改变**，不是纯"更稳"。legacy 每条查询重掷、缓存每快照定一次； + 若 BE 存在"仅当混格式表 scan 级取到 ORC 时才触发"的潜伏 bug，其表现会从"间歇"变成"按快照恒发/恒不发"。 + correctness 不变，但把它当**可测试性变化**记账，不当卖点。 +- **时间旅行**：键用 `table.currentSnapshot().snapshotId()`（= 推断真正读的快照），非 handle 的 pin 快照； + 与 legacy「格式恒从 currentSnapshot 推」一致；不同 pin 的时旅查询正确共享同一 currentSnapshot 派生的格式。 +- **失败可重试**：抛异常的 loader 不入缓存 → 瞬时失败下条重试，不粘住错误默认。 + +## Implementation Plan（最小改动，逐文件） + +1. 新建 `IcebergFormatCache.java`（≈ 130 行，镜像 `IcebergPartitionCache`，值 `String`）。 +2. `IcebergWriterHelper.java`：+ import `Snapshot`、`TableIdentifier`；加 cache-aware `getFileFormat` 重载 + + `resolveFileFormatName` 重载 + 拆 `inferFirstDataFileFormat`（透传）/`inferFileFormatFromDataFiles`（吞→委派）。 +3. `IcebergScanPlanProvider.java`：+ 字段 + ctor 参 + 改 1330 调用点。 +4. `IcebergConnector.java`：+ 字段/构造/三个 invalidate/getScanPlanProvider 传参/`formatCacheForTest`。 + +## Test Plan + +- **单测 `IcebergFormatCacheTest`**（镜像 `IcebergPartitionCacheTest`）：TTL 内命中同键 loadCount=1、 + 不同 snapshotId 不同键、`ttl<=0`/负值关缓存、`invalidate`(清该表所有快照)/`invalidateDb`/`invalidateAll`、 + 抛异常 loader 不入缓存（失败可重试）。 +- **度量守门（集成）** 在 `IcebergWriterHelperTest`（或 `IcebergPartitionUtilsTest` 同款 InMemoryCatalog 真表）： + 建**无 `write.format.default` 属性**的真表 → 重复 `getFileFormat(table, id, cache)` N 次 → + `cache.loadCountForTest()==1`（跨查询恰 1 次远端推断），并与无缓存 `getFileFormat(table)` 结果 parity。 + **MUTATION**：不把 cache 接进调用点 → 每次重扫 → loadCount>1 → 红。 + 另断言**属性已带格式**的表根本不进缓存（`size()==0`，属性路径不缓存）。 + - **R3（红队可选断言）**：首文件为不支持格式（如构造一条 `FileFormat.AVRO` 的 DataFile 条目）时，`getFileFormat` + 在 `getOrLoad` **之外**的 orc/parquet 映射处每次重抛（parity），但推断只跑一次 → 两次调用都抛且 `loadCount==1`。 +- **连接器 gate + 失效 `IcebergConnectorCacheTest`**：plain/vended/session=user 三目录 `formatCacheForTest()` + 均非 null（无 gate）；REFRESH TABLE/DB/CATALOG 逐级清空。 +- 全 iceberg 模块 UT 绿（parity，禁回归）+ checkstyle 绿。 + +## Risk + +- **低**：连接器自包含改动，无 fe-core、无 thrift、无 BE。唯一行为差异是"混格式表 scan 级格式从任意→每快照稳定"， + 已论证为 parity-中性。失败不缓存守住可重试。V1/V2 scanner 选择值语义不变（仍是表真实格式，非 jni）。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-03-format-inference-cache-summary.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-03-format-inference-cache-summary.md new file mode 100644 index 00000000000000..66c0438cda4052 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-03-format-inference-cache-summary.md @@ -0,0 +1,58 @@ +# FIX-PERF-03 — Summary（`file_format_type` 兜底整表 planFiles 跨查询 memoize） + +> 权威设计见 [`FIX-PERF-03-format-inference-cache-design.md`](./FIX-PERF-03-format-inference-cache-design.md)。本文件只记落地结果。 +> commit `0b96f2e6c78`（`[perf]`，iceberg 自包含）。基线 HEAD `cab6e1ed50e`（PERF-01/02 已合）。 + +## Problem + +每条 `SELECT`/`EXPLAIN` 在 scan-node 初始化期算 scan 级 `file_format_type`（`getScanNodeProperties:1350` → +`IcebergWriterHelper.getFileFormat`）。表属性无 `write-format` 且无 `write.format.default`（迁移表 + 从不显式设该属性的表） +时退化成 `inferFileFormatFromDataFiles → table.newScan().planFiles()` —— **无过滤整表 manifest 扫描**（远端 IO， +`ParallelIterable` 激进 fan-out），只为读第一个数据文件的格式。node 级已 memoize 故查询内 1 次，但**跨查询零缓存**： +大表每查询数百 ms~秒级远端 IO ×QPS。这是 #64134 兜底在新框架以 per-query 乘数复活（C2/C11 同一重操作两视角）。 + +## Fix（连接器侧，零 fe-core 改动） + +- **新增 `IcebergFormatCache`**（挂 `IcebergConnector`）：键 `(TableIdentifier, currentSnapshotId)`，值 = 推断出的 + 格式名 `String`。快照不可变 ⇒ 首文件格式是键的纯函数；snapshotId 由 `IcebergLatestSnapshotCache` 跨查询稳定。 + 只缓存**推断兜底**这条，两个属性探测保持 cheap 不缓存。 +- **无凭证 gate**（同 PERF-02 partition cache）：值是纯元数据（格式名，无 FileIO/token）→ 所有目录无条件构造 + （含 session=user / REST vended），仅 TTL 控。REFRESH TABLE/DB/CATALOG 逐级清空。 +- **`IcebergWriterHelper`**：加 cache-aware `getFileFormat(Table, TableIdentifier, IcebergFormatCache)` 重载 + + `resolveFileFormatName` 重载；拆 `inferFirstDataFileFormat`(**透传异常**，`close()` 的 checked `IOException` + 包成 `UncheckedIOException`) / `inferFileFormatFromDataFiles`(吞→委派，写路径保留)。orc/parquet 映射与其 + unsupported-format throw 在 `getOrLoad` **之外** → 缓存永不存"会抛"的值。 +- **`IcebergScanPlanProvider`**：加 nullable `formatCache` 字段 + 7 参 ctor（6 参 ctor 委派传 null，离线测试与 + 既有调用全走 null 保持原行为）；`getScanNodeProperties:1350` 非系统表分支改用 cache-aware 重载。 +- **写路径 7 个 `getFileFormat(Table)` 调用点不动**（PERF-07/C20 范围）。 + +## 路线决定（复核否决"从 planScan 反推"） + +`FileQueryScanNode.createScanRangeLocations` **先** `getFileFormatType`（line 325）**后** `getSplits/planScan` +（line 422）：格式在数据扫描前就要给出，无法复用 planScan 枚举；且推断是无过滤、planScan 带谓词，剪空时反推破 +parity。改时序须动 fe-core（违反铁律）→ 退回 memoize（正是设计预判）。 + +## Parity + +- 同 currentSnapshot 推断（非 handle time-travel pin），与 legacy `IcebergUtils.getFileFormat` 一致；时旅查询共享。 +- **失败不缓存**：loader 抛出（unchecked）→ `MetaCacheEntry` 不 put → 下条查询重试（legacy 吞→parquet 的可重试性保留）。 +- **混格式表**：scan 级格式从"每查询任意"变"每快照确定"——correctness 中性（per-file 格式仍逐 range），记为 + **可复现性变化**（红队 R2）：某"仅 scan 级 ORC 才触发"的潜伏 BE bug 会从间歇变每快照恒发/恒不发。 +- scan 级值仍只选 BE V1/V2 scanner，语义不变。 + +## Tests + +- **单测 `IcebergFormatCacheTest`**（8）：TTL 内命中同键 loadCount=1、不同 snapshotId 不同键、`ttl<=0`/负值关、 + invalidate/DB/All、**失败不入缓存+可重试**。 +- **度量守门（集成）** `IcebergWriterHelperTest`（真 InMemoryCatalog 表 + 真数据文件）：无格式属性表重复 + `getFileFormat(table,id,cache)` 3 次 → `loadCountForTest()==1`+`size()==1`，与无缓存 parity；**属性表不进缓存** + （`size()==0`）；null cache 存活；**avro 首文件**推断一次但每次重抛（R3，loadCount==1）。 +- **连接器 gate + 失效** `IcebergConnectorCacheTest`（+2）：plain/vended/session 三目录 `formatCacheForTest()` 非 null； + REFRESH TABLE/DB/CATALOG 逐级清空。 + +## Result + +- 全 iceberg 模块 **957 pass / 0 fail / 1 skip**（`install -am`），checkstyle 绿，0 回归。 +- 减负：无格式属性表每查询整表格式推断从"每查询一遍"→"每快照一遍"（跨查询命中）。metastore/对象存储 IO 除以 QPS-over-TTL。 +- 可复用产物：`IcebergFormatCache`（第三块 `(TableIdentifier[,snapshotId])` MetaCacheEntry 基建，值 `String`）； + 「值纯元数据→无 gate」判据第三次印证；「映射/抛点放 getOrLoad 之外，缓存只存 loader 原始输出」的失败不缓存范式。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index 3fd6690d47d19a..6a37a042c55c13 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -45,3 +45,16 @@ - **度量守门（新）**：`IcebergPartitionUtilsTest.partitionScanIsCachedAcrossRepeatsAndConsumersAtSameSnapshot` —— 真分区表重复 `buildMvccPartitionView` + `listPartitions` 同快照 → `loadCountForTest()==1`。另 `IcebergPartitionCacheTest`（含 `ValidationException` 透传+不缓存）、连接器 gate/失效诸测。 - **结果**：全 iceberg 模块 **943 pass / 0 fail / 1 skip**，checkstyle 绿。summary 见 `designs/FIX-PERF-02-partition-view-cache-summary.md`。0 回归（无需改任何现有测试）。 - **下一步**：见 HANDOFF —— PERF-03（#64134 planFiles 兜底复活：`file_format_type` 无 write-format 时走整表 planFiles 反推格式 → memoize / 从枚举反推）。 + +--- + +## 2026-07-18 — session 3：PERF-03 实现 + 全绿（commit `0b96f2e6c78`） + +- **复核确认审计 C2/C11（同一重操作两视角）**：`getScanNodeProperties:1350 → IcebergWriterHelper.getFileFormat → resolveFileFormatName → inferFileFormatFromDataFiles → table.newScan().planFiles()`（无过滤整表 manifest 扫描）。门槛=表属性无 `write-format` 且无 `write.format.default`。fe-core 侧 `cachedPropertiesResult` 已 node-memoize ⇒ 查询内 1 次，但跨查询零缓存 ⇒ 每查询/EXPLAIN 一遍整表扫描（大表数百 ms~秒 ×QPS）。 +- **路线否决（复核推翻 HANDOFF 原倾向 b）**：HANDOFF 倾向"从 planScan 枚举反推格式"（消除而非缓存），复核**否决**——`FileQueryScanNode.createScanRangeLocations` **先** `getFileFormatType`（line 325 → 触发本重操作）**后** `getSplits/planScan`（line 422）：格式须先于数据扫描给出，planScan 尚未跑；且推断是无过滤、planScan 带谓词，谓词剪空所有文件时反推拿不到格式破 legacy parity；改时序须动 fe-core（违反"fe-core 源只出不进"铁律）。**正是 HANDOFF 预判的"两路径独立触发 → 退回 (a) memoize"**。 +- **设计红队（独立 agent 对抗）**：7 项攻击（时序/键 soundness/无 gate/失败粘住/混格式&空表/失效完整/写路径不缓存）**全部未击穿**，判"根本可靠"。采纳两点：R1 loader 只能抛 unchecked（`close()` 的 `IOException` 包 `UncheckedIOException`）；R2 混格式表"格式变确定"如实记为可复现性变化非卖点。R3 加 avro 首文件重抛断言。 +- **实现（连接器侧，一个 `[perf]` commit，零 fe-core 改动）**：新增 `IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、值格式名 `String`、**无凭证 gate**）；`IcebergWriterHelper` 加 cache-aware `getFileFormat` 重载 + `resolveFileFormatName` 重载，拆 `inferFirstDataFileFormat`(透传异常)/`inferFileFormatFromDataFiles`(吞→委派)，orc/parquet 映射与其 unsupported throw 保持在 `getOrLoad` 之外；`IcebergScanPlanProvider` 加 nullable `formatCache` 字段 + 7 参 ctor（6 参委派 null）+ 改 `getScanNodeProperties` 调用点；`IcebergConnector` 无条件构造 + 传 provider + 三失效钩子 + `formatCacheForTest`。**写路径 7 个 `getFileFormat(Table)` 调用点不动**（留 PERF-07/C20）。 +- **度量守门（新）**：`IcebergWriterHelperTest`（真 InMemoryCatalog 表 + 真数据文件）无格式属性表重复 `getFileFormat(table,id,cache)` → `loadCountForTest()==1`+`size()==1`+与无缓存 parity；属性表 `size()==0`；null cache 存活；avro 首文件推断一次但每次重抛。`IcebergFormatCacheTest`（8：TTL/关/失效/失败不缓存可重试）。`IcebergConnectorCacheTest`（+2：三目录无 gate + REFRESH 失效）。 +- **结果**：全 iceberg 模块 **957 pass / 0 fail / 1 skip**（`install -am`），checkstyle 绿，0 回归（无需改任何现有测试）。summary 见 `designs/FIX-PERF-03-format-inference-cache-summary.md`。 +- **踩坑/印证**：InMemoryCatalog `newAppend().commit()` 后同一 table 对象 `currentSnapshot()` 会刷新（度量守门测试 ORC+loadCount==1 通过即证），无需像 `dayPartitionedTable` 那样 reload；但 reload 仍是更稳的既有范式。 +- **下一步**：见 HANDOFF —— PERF-04（IcebergManifestCache 两条旁路接回：C17 streaming 大表踢出 cache + C18 COUNT(*) 下推绕过 cache）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index b56adadc27047f..9da361b793c947 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -17,7 +17,7 @@ |---|---|---|---|---|---|---| | PERF-01 | P0 | C1 C4 C6 C10 C16 | 一次规划 3~7 次远程 loadTable → 胖 handle(查询内单实例)+ 跨查询 IcebergTableCache(挂 Connector);~~convertPredicate 收窄~~已删(红队证伪) | — | ✅ 完成 | `484f0e0c125` | | PERF-02 | P0 | C7 C22 C23 | 分区视图每查询重扫 PARTITIONS 元数据表 → `(table,snapshotId)` 缓存(连接器侧,无凭证 gate);MTMV refresh pin 判定为多余(靠 latestSnapshotCache 稳定快照坍缩,不改 fe-core) | 与 01 共享快照 pin 机制 | ✅ 完成 | `518d0599cbf` | -| PERF-03 | P0 | C2 C11 | #64134 复活：`file_format_type` 兜底走整表 planFiles → memoize / 从枚举反推 | 受益于 01 的失效收窄（消第二次） | ⏳ | | +| PERF-03 | P0 | C2 C11 | #64134 复活：`file_format_type` 兜底走整表 planFiles → 跨查询 `(table,snapshotId)` memoize(连接器侧,无 gate);~~从枚举反推~~已否决(getFileFormatType 早于 planScan + 无过滤 vs 带谓词破 parity) | 与 01/02 共享快照 pin | ✅ 完成 | `0b96f2e6c78` | | PERF-04 | P1 | C17 C18 | streaming / COUNT(*) 下推旁路 IcebergManifestCache → 两条旁路接回 | — | ⏳ | | | PERF-05 | P1 | C9 | information_schema.tables 循环内每表 loadTable（只为拿 comment） → 随表缓存 / 惰性取 | — | ⏳ | | | PERF-06 | P1 | C3 | REST vended-cred 每 data/delete file 重建 StorageProperties+Configuration → scan 级 memo | — | ⏳ | | @@ -43,10 +43,11 @@ - **修复方向**：按 `(TableIdentifier, snapshotId)` 缓存分区视图（pin 在 `beginQuerySnapshot` 后已在 handle 上），挂 `fe-connector-cache`；MTMV 侧在 `MTMVRefreshContext` 加 refresh 级 `MvccSnapshot` pin。 - **依赖**：与 PERF-01 共享 `(table, snapshotId)` 快照 pin —— 先做 01 立住模式，02 复用。 -### [ ] PERF-03 — 簇2：#64134 planFiles 兜底复活（C2 C11） -- **病灶**：`getScanNodeProperties:1311 → IcebergWriterHelper.getFileFormat:265 → resolveFileFormatName:277 → inferFileFormatFromDataFiles:287 → table.newScan().planFiles():291`（无过滤整表 manifest 扫描）。门槛=表属性无 `write-format` 且无 `write.format.default`（含迁移表 + 任何写引擎从不显式设该属性的表）。纯冗余：planScan 自己的 planFiles 枚举里每个 `dataFile.format()` 都有，为拿"第一个文件格式"专门再扫全表。乘数=每次 properties 计算 1 次（无谓词 1/查询、带谓词 2/查询，叠 C1）。 -- **修复方向**：按 `(table UUID, snapshotId)` memoize 解析结果（快照不变 ⇒ 格式不变），或从 split 枚举结果反推 scan 级 format 传下去；配合 PERF-01 的失效收窄消掉第二次。 -- **依赖**：PERF-01 的 convertPredicate 收窄消除第二次计算。 +### [x] PERF-03 — 簇2：#64134 planFiles 兜底复活（C2 C11） · ✅ `0b96f2e6c78` +- **病灶**：`getScanNodeProperties → IcebergWriterHelper.getFileFormat → resolveFileFormatName → inferFileFormatFromDataFiles → table.newScan().planFiles()`（无过滤整表 manifest 扫描）。门槛=表属性无 `write-format` 且无 `write.format.default`（迁移表 + 任何写引擎从不显式设该属性的表）。node 级已 memoize ⇒ 查询内 1 次，但跨查询零缓存 ⇒ 每查询/EXPLAIN 一遍整表扫描。 +- **落地**：新增连接器侧 `IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、值格式名 `String`、**无凭证 gate**），只缓存推断兜底（属性探测不缓存），失败不入缓存（loader 透传 unchecked），映射/抛点在 `getOrLoad` 之外。写路径 7 调用点不动（留 PERF-07）。 +- **~~从枚举反推~~已否决**：`createScanRangeLocations` 先 `getFileFormatType`(line 325) 后 `getSplits/planScan`(line 422)，格式须先于数据扫描给出；且推断无过滤 vs planScan 带谓词，剪空时反推破 parity；改时序须动 fe-core（违反铁律）。 +- **收益**：无格式属性表每查询整表推断 → 每快照一遍（跨查询命中）。度量守门 `loadCountForTest()==1`。全模块 957 UT 绿。 --- From a627fea81a3b1c157f01f73883def607d4fb6e3b Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 09:29:27 +0800 Subject: [PATCH 10/47] [perf](catalog) fe-connector-iceberg: reconnect the manifest cache to streaming + COUNT(*) planning, lazily (PERF-04) Root cause: the opt-in IcebergManifestCache (meta.cache.iceberg.manifest.enable, default off) was consulted only on the synchronous planning path (planFileScanTaskWithManifestCache). Two paths bypassed it: (C17) the streaming split path for tables >= num_files_in_batch_mode -- the OOM-protection lazy path, which is exactly the large tables the cache targets -- read scan.planFiles() (SDK, uncached); (C18) COUNT(*) pushdown read scan.planFiles() just to grab one placeholder file (the count itself comes from the snapshot summary), while iceberg's ParallelIterable eagerly submitted every manifest reader. On a cache-enabled catalog the cache delivered 0% hit rate on its target tables. Legacy contrast (verified against 6fef25709d3^ fe-core IcebergScanNode): batch mode DID honor the cache -- but the cache-on path MATERIALIZED the full task list (no OOM protection); OOM safety existed only with the cache off. The SPI cutover traded the cache for unconditional streaming (OOM-safe, uncached). Rather than restore legacy (reintroducing the OOM risk) or leave it, this makes the cache-backed planning itself LAZY so both paths get cache hits AND stay bounded. Fix (connector-side, no fe-core change): extract cacheBackedFileScanTasks(), a lazy CloseableIterable shared by all three consumers. Phase 1 (eager) reads the delete manifests through the cache and builds the DeleteFileIndex -- required before any data task and bounded like the SDK planFiles(), which also reads all delete manifests up front. Phase 2 (lazy) is a flat-map iterator over the data manifests, reading each through the cache on demand and yielding pruned whole-file BaseFileScanTasks WITHOUT materializing the list (peak heap = delete index + one manifest's files + the split queue). - planFileScanTaskWithManifestCache now materializes that iterable (byte-identical to before; the sync/small-table path keeps the per-table heuristic split size). - streamSplits, when the cache is enabled, feeds the lazy iterable to the existing IcebergStreamingSplitSource at a fixed slice size (unchanged for big tables) -- now cache-hitting and still OOM-safe. - planCountPushdown, when enabled, iterates the lazy iterable and takes the FIRST file (lazy early stop, no ParallelIterable fan-out). The batch/stream DECISION (streamingSplitEstimate) is unchanged. A cache-read failure records it and falls back to scan.planFiles() (catch Exception, since the cache rethrows failures as RuntimeException). statsQueryId is nullable: streaming passes null (no-stats overload) because its Phase 2 runs on the engine pump thread while Phase 1 ran on the caller -- tallying the per-query ScanStats from two threads would race; sync/count are single-threaded and keep stats. Parity: the lazy path reuses the exact per-file prune/task-construction of the old materialized path (which already claims legacy planFiles parity), so no new planning is invented. The COUNT placeholder file may differ from the SDK path's first file (ParallelIterable order is non-deterministic) but the count (snapshot summary) is identical and BE ignores the file. Big-table streaming now populates the cache (its purpose; capacity-bounded, no OOM). Tests: streaming cache-parity vs SDK + cache consumed; partition-prune parity; flat-map across multiple data manifests; COUNT cache count-parity + lazy early stop (reads < all manifests); empty null-snapshot COUNT guard. Sync cache-path tests stay green. Full iceberg module UT: 962 run / 0 fail / 0 error / 1 skip; checkstyle 0 violations. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../iceberg/IcebergScanPlanProvider.java | 261 +++++++++++++++--- .../iceberg/IcebergScanPlanProviderTest.java | 114 ++++++++ 2 files changed, 329 insertions(+), 46 deletions(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index 093f8a518fdf40..d5124dcd9c4995 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -95,6 +95,7 @@ import java.util.Collections; import java.util.HashMap; import java.util.HashSet; +import java.util.Iterator; import java.util.LinkedHashMap; import java.util.List; import java.util.Locale; @@ -497,11 +498,34 @@ public ConnectorSplitSource streamSplits(ConnectorSession session, ConnectorTabl long fileSplitSize = sessionLong(session, FILE_SPLIT_SIZE, 0L); long sliceSize = fileSplitSize > 0 ? fileSplitSize : sessionLong(session, MAX_FILE_SPLIT_SIZE, DEFAULT_MAX_FILE_SPLIT_SIZE); - CloseableIterable tasks = TableScanUtil.splitFiles(scan.planFiles(), sliceSize); + CloseableIterable tasks = streamingFileScanTasks(scan, session, table, filter, sliceSize); return new IcebergStreamingSplitSource(tasks, table, formatVersion, partitioned, orderedPartitionKeys, zone, vendedToken, sliceSize, iceHandle.getRewriteFileScope()); } + /** + * The streaming source's whole-file enumeration, byte-offset-split at {@code sliceSize}. PERF-04 (C17): when + * the manifest cache is enabled, read manifests THROUGH THE CACHE via the lazy {@link #cacheBackedFileScanTasks} + * (no-stats overload — Phase 2 iterates on the engine pump thread, see there) so a big streaming scan finally + * hits the cache while staying lazy (OOM-safe). An eager Phase-1 cache failure records it and falls back to the + * SDK {@code planFiles()} path (mirrors {@link #planFileScanTask}); a later lazy failure surfaces through the + * streaming source's {@code hasNext}. Cache disabled -> the SDK path, byte-unchanged. + */ + private CloseableIterable streamingFileScanTasks(TableScan scan, ConnectorSession session, + Table table, Optional filter, long sliceSize) { + if (isManifestCacheEnabled()) { + try { + return TableScanUtil.splitFiles( + cacheBackedFileScanTasks(scan, session, table, filter, null), sliceSize); + } catch (Exception e) { + LOG.warn("Iceberg streaming plan with manifest cache failed, falling back to SDK scan: {}", + e.getMessage(), e); + manifestCache.recordFailure(session.getQueryId()); + } + } + return TableScanUtil.splitFiles(scan.planFiles(), sliceSize); + } + /** * Lazy {@link ConnectorSplitSource} over an iceberg scan's byte-offset-split {@link FileScanTask}s: maps each * task to an {@link IcebergScanRange} on demand (via {@link #buildRangeForTask}) so the engine can pump them @@ -632,7 +656,7 @@ private List planScanInternal( long realCount = getCountFromSnapshot(scan, session); if (realCount >= 0) { return planCountPushdown(table, scan, realCount, formatVersion, partitioned, - orderedPartitionKeys, zone, vendedToken); + orderedPartitionKeys, zone, vendedToken, session, filter); } } @@ -1127,8 +1151,8 @@ private static Schema pinnedSchema(Table table, IcebergTableHandle handle) { */ private List planCountPushdown(Table table, TableScan scan, long realCount, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, - Map vendedToken) { - try (CloseableIterable tasks = scan.planFiles()) { + Map vendedToken, ConnectorSession session, Optional filter) { + try (CloseableIterable tasks = countPushdownFileScanTasks(scan, session, table, filter)) { for (FileScanTask task : tasks) { // targetSplitSize = -1: the count-pushdown collapse emits a single range, so its scheduling // weight is irrelevant → PluginDrivenSplit keeps SplitWeight.standard(). @@ -1142,6 +1166,31 @@ private List planCountPushdown(Table table, TableScan scan, return Collections.emptyList(); } + /** + * The COUNT(*)-pushdown placeholder enumeration: only the FIRST surviving file is consumed (BE serves the + * count from {@code table_level_row_count} and never reads the file). PERF-04 (C18): when the manifest cache is + * enabled, read through the lazy {@link #cacheBackedFileScanTasks} (stats overload — this runs on the single + * planning thread) so the manifest reads are cache hits and, being lazy, stop at the first file's manifest + * instead of the SDK {@code planFiles()}'s {@code ParallelIterable} eagerly submitting every manifest reader. + * An eager cache failure falls back to the SDK path (mirrors {@link #planFileScanTask}). The first surviving + * (pruned) file may differ from the SDK path's first file (its {@code ParallelIterable} order is + * non-deterministic), but the count is identical (from the snapshot summary) and BE ignores the file. Cache + * disabled -> the SDK path, byte-unchanged. + */ + private CloseableIterable countPushdownFileScanTasks(TableScan scan, ConnectorSession session, + Table table, Optional filter) { + if (isManifestCacheEnabled()) { + try { + return cacheBackedFileScanTasks(scan, session, table, filter, session.getQueryId()); + } catch (Exception e) { + LOG.warn("Iceberg count-pushdown plan with manifest cache failed, falling back to SDK scan: {}", + e.getMessage(), e); + manifestCache.recordFailure(session.getQueryId()); + } + } + return scan.planFiles(); + } + /** * Build the BE-ready {@link IcebergScanRange} for one {@link FileScanTask}, mirroring legacy * {@code IcebergScanNode.createIcebergSplit} + {@code setIcebergParams}: the file path/offset/size, the @@ -1811,24 +1860,62 @@ private SplitPlan planFileScanTask(TableScan scan, ConnectorSession session, Tab } /** - * Manifest-level planning that consumes {@link IcebergManifestCache}, ported faithfully from legacy - * {@code IcebergScanNode.planFileScanTaskWithManifestCache}. It reconstructs iceberg's own planning: - * partition-prune manifests with a {@link ManifestEvaluator}, read each surviving manifest's data/delete - * files THROUGH THE CACHE, then per data file apply the {@link InclusiveMetricsEvaluator} (file-stats - * pruning) + {@link ResidualEvaluator} (partition residual) and attach its deletes via a - * {@link DeleteFileIndex}. The resulting {@link FileScanTask}s are byte-offset-split exactly like - * {@link #splitFiles}, so the downstream {@code buildRange} (T03-T07) is unchanged. The predicate / - * metrics / schema use the table's CURRENT schema (legacy parity). + * Synchronous (below-batch-threshold) manifest-cache planning: materialize the shared lazy + * {@link #cacheBackedFileScanTasks} enumeration into a list, because {@link #determineTargetFileSplitSize} + * (the per-table heuristic that gives small tables good BE parallelism) needs the whole task list. Byte + * identical to the pre-PERF-04 body; the streaming ({@link #streamSplits}) and COUNT(*) + * ({@link #planCountPushdown}) paths consume the same enumeration LAZILY instead (bounded FE heap). Uses the + * stats-tallying cache overload — this path runs on the single planning thread. The predicate / metrics / + * schema use the table's CURRENT schema (legacy parity). */ private SplitPlan planFileScanTaskWithManifestCache(TableScan scan, ConnectorSession session, Table table, Optional filter) throws IOException { + // Null-safe queryId (offline tests pass a null session): a null id selects the no-stats cache overload, + // matching the pre-PERF-04 body, which read scan.snapshot() and returned early for an empty table BEFORE + // ever calling session.getQueryId(). + String statsQueryId = session != null ? session.getQueryId() : null; + List tasks = new ArrayList<>(); + try (CloseableIterable whole = + cacheBackedFileScanTasks(scan, session, table, filter, statsQueryId)) { + for (FileScanTask task : whole) { + tasks.add(task); + } + } + long targetSplitSize = determineTargetFileSplitSize(tasks, session); + return new SplitPlan( + TableScanUtil.splitFiles(CloseableIterable.withNoopClose(tasks), targetSplitSize), targetSplitSize); + } + + /** + * PERF-04: the lazy, cache-backed {@link FileScanTask} enumeration shared by the synchronous + * ({@link #planFileScanTaskWithManifestCache}), streaming ({@link #streamSplits}), and COUNT(*) + * ({@link #planCountPushdown}) paths whenever {@link #isManifestCacheEnabled()}. Ported faithfully from legacy + * {@code IcebergScanNode.planFileScanTaskWithManifestCache}: partition-prune manifests with a + * {@link ManifestEvaluator}, read each surviving manifest's data/delete files THROUGH THE CACHE, then per data + * file apply the {@link InclusiveMetricsEvaluator} (file-stats prune) + {@link ResidualEvaluator} (partition + * residual) and attach its deletes via a {@link DeleteFileIndex}. Produces WHOLE-FILE + * {@link BaseFileScanTask}s (callers byte-offset-split via {@link TableScanUtil#splitFiles}). + * + *

Phase 1 is eager (delete-manifest reads + {@link DeleteFileIndex} build): a data file's deletes + * need the full index before any data task can be produced, and running it here (on the caller's thread) lets + * the streaming/count callers catch a cache failure and fall back to the SDK path. Bounded like the SDK + * {@code planFiles()} (which also reads all delete manifests up front); delete manifests are far fewer than + * data manifests. Phase 2 is lazy: the returned iterable's iterator flat-maps the matching data + * manifests, yielding one surviving task at a time WITHOUT materializing the list, so a million-file streaming + * scan keeps FE heap bounded (peak = the delete index + one manifest's files + the split queue). + * + *

{@code statsQueryId} is nullable: a non-null id tallies cache hits/misses under that query (the + * single-threaded synchronous + COUNT paths); {@code null} selects the no-stats overload for the STREAMING + * path, whose Phase 2 runs on the engine pump thread while Phase 1 ran on the calling thread — tallying the + * per-query {@code ScanStats} counters from two threads would race (the cache documents them as + * single-thread-per-query), and streaming reports no manifest-cache stats today anyway. + */ + private CloseableIterable cacheBackedFileScanTasks(TableScan scan, + ConnectorSession session, Table table, Optional filter, String statsQueryId) { Snapshot snapshot = scan.snapshot(); if (snapshot == null) { - return new SplitPlan(CloseableIterable.withNoopClose(Collections.emptyList()), -1); + return CloseableIterable.withNoopClose(Collections.emptyList()); } - // Stable per-statement key so VERBOSE EXPLAIN (rendered on a different, transient provider instance) can - // report THIS scan's manifest-cache hits/misses via the shared per-catalog cache. - String queryId = session.getQueryId(); Expression filterExpr = combineFilter(filter, table, session); Map specsById = table.specs(); boolean caseSensitive = true; @@ -1838,8 +1925,9 @@ private SplitPlan planFileScanTaskWithManifestCache(TableScan scan, ResidualEvaluator.of(spec, filterExpr, caseSensitive))); InclusiveMetricsEvaluator metricsEvaluator = new InclusiveMetricsEvaluator(table.schema(), filterExpr, caseSensitive); + String schemaJson = SchemaParser.toJson(table.schema()); - // Phase 1: partition-prune + cache-load delete manifests into a flat delete-file list. + // Phase 1 (eager): partition-prune + cache-load delete manifests into the delete-file index. List deleteFiles = new ArrayList<>(); for (ManifestFile manifest : snapshot.deleteManifests(table.io())) { if (manifest.content() != ManifestContent.DELETES) { @@ -1852,50 +1940,131 @@ private SplitPlan planFileScanTaskWithManifestCache(TableScan scan, if (!ManifestEvaluator.forPartitionFilter(filterExpr, spec, caseSensitive).eval(manifest)) { continue; } - deleteFiles.addAll(manifestCache.getManifestCacheValue(manifest, table, queryId).getDeleteFiles()); + deleteFiles.addAll(manifestCacheGet(manifest, table, statsQueryId).getDeleteFiles()); } DeleteFileIndex deleteIndex = DeleteFileIndex.builderFor(deleteFiles) .specsById(specsById) .caseSensitive(caseSensitive) .build(); - // Phase 2: partition-prune + cache-load data manifests, then file-level prune + attach deletes. - List tasks = new ArrayList<>(); - try (CloseableIterable dataManifests = - getMatchingManifest(snapshot.dataManifests(table.io()), specsById, filterExpr)) { - for (ManifestFile manifest : dataManifests) { - if (manifest.content() != ManifestContent.DATA) { - continue; - } - PartitionSpec spec = specsById.get(manifest.partitionSpecId()); - if (spec == null) { - continue; - } - ResidualEvaluator residualEvaluator = residualEvaluators.get(manifest.partitionSpecId()); - if (residualEvaluator == null) { - continue; - } - ManifestCacheValue value = manifestCache.getManifestCacheValue(manifest, table, queryId); - for (DataFile dataFile : value.getDataFiles()) { + // Phase 2 (lazy): flat-map the matching data manifests, read through the cache on demand. + CloseableIterable dataManifests = + getMatchingManifest(snapshot.dataManifests(table.io()), specsById, filterExpr); + return new CloseableIterable() { + @Override + public CloseableIterator iterator() { + return new ManifestCacheFileScanTaskIterator(dataManifests.iterator(), table, specsById, + residualEvaluators, metricsEvaluator, deleteIndex, schemaJson, statsQueryId); + } + + @Override + public void close() throws IOException { + dataManifests.close(); + } + }; + } + + /** Dispatch a manifest read to the stats-tallying or no-stats cache overload (see cacheBackedFileScanTasks). */ + private ManifestCacheValue manifestCacheGet(ManifestFile manifest, Table table, String statsQueryId) { + return statsQueryId != null + ? manifestCache.getManifestCacheValue(manifest, table, statsQueryId) + : manifestCache.getManifestCacheValue(manifest, table); + } + + /** + * Lazy flat-map iterator behind {@link #cacheBackedFileScanTasks}: walks the matching data manifests, reads + * each through the manifest cache on demand, and yields the surviving (metrics + residual pruned) + * {@link BaseFileScanTask}s one file at a time so the consumer never holds the whole table's task list. Not + * thread-safe: single-pass, driven by ONE consumer (the sync materialize loop, the streaming pump, or the + * COUNT take-first). {@code schemaJson}/{@code currentSpecJson} are hoisted loop invariants (constant per + * table / per manifest spec), byte-identical to the pre-PERF-04 per-file computation. + */ + private final class ManifestCacheFileScanTaskIterator implements CloseableIterator { + private final CloseableIterator manifestIt; + private final Table table; + private final Map specsById; + private final Map residualEvaluators; + private final InclusiveMetricsEvaluator metricsEvaluator; + private final DeleteFileIndex deleteIndex; + private final String schemaJson; + private final String statsQueryId; + + private Iterator dataFileIt; + private ResidualEvaluator currentResidual; + private String currentSpecJson; + private FileScanTask next; + + ManifestCacheFileScanTaskIterator(CloseableIterator manifestIt, Table table, + Map specsById, Map residualEvaluators, + InclusiveMetricsEvaluator metricsEvaluator, DeleteFileIndex deleteIndex, String schemaJson, + String statsQueryId) { + this.manifestIt = manifestIt; + this.table = table; + this.specsById = specsById; + this.residualEvaluators = residualEvaluators; + this.metricsEvaluator = metricsEvaluator; + this.deleteIndex = deleteIndex; + this.schemaJson = schemaJson; + this.statsQueryId = statsQueryId; + } + + @Override + public boolean hasNext() { + advance(); + return next != null; + } + + @Override + public FileScanTask next() { + advance(); + if (next == null) { + throw new NoSuchElementException(); + } + FileScanTask result = next; + next = null; + return result; + } + + // Fill `next` with the next surviving task, draining the current data manifest's files and advancing to + // further data manifests as needed. Per-file logic mirrors the pre-PERF-04 materialized Phase 2 exactly. + private void advance() { + while (next == null) { + if (dataFileIt != null && dataFileIt.hasNext()) { + DataFile dataFile = dataFileIt.next(); if (!metricsEvaluator.eval(dataFile)) { continue; } - if (residualEvaluator.residualFor(dataFile.partition()).equals(Expressions.alwaysFalse())) { + if (currentResidual.residualFor(dataFile.partition()).equals(Expressions.alwaysFalse())) { continue; } DeleteFile[] deletes = deleteIndex.forDataFile(dataFile.dataSequenceNumber(), dataFile); - tasks.add(new BaseFileScanTask( - dataFile, - deletes, - SchemaParser.toJson(table.schema()), - PartitionSpecParser.toJson(spec), - residualEvaluator)); + next = new BaseFileScanTask(dataFile, deletes, schemaJson, currentSpecJson, currentResidual); + return; } + if (!manifestIt.hasNext()) { + return; + } + ManifestFile manifest = manifestIt.next(); + if (manifest.content() != ManifestContent.DATA) { + dataFileIt = null; + continue; + } + PartitionSpec spec = specsById.get(manifest.partitionSpecId()); + ResidualEvaluator residual = residualEvaluators.get(manifest.partitionSpecId()); + if (spec == null || residual == null) { + dataFileIt = null; + continue; + } + currentResidual = residual; + currentSpecJson = PartitionSpecParser.toJson(spec); + dataFileIt = manifestCacheGet(manifest, table, statsQueryId).getDataFiles().iterator(); } } - long targetSplitSize = determineTargetFileSplitSize(tasks, session); - return new SplitPlan( - TableScanUtil.splitFiles(CloseableIterable.withNoopClose(tasks), targetSplitSize), targetSplitSize); + + @Override + public void close() throws IOException { + manifestIt.close(); + } } /** diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index af66ca29b6acd0..41c038b0780c86 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -2008,6 +2008,120 @@ public void planScanManifestCachePrunesPartitionLikeSdk() { Assertions.assertTrue(manifest.get(0).getPath().get().endsWith("p1.parquet")); } + // --- PERF-04: streaming (C17) + COUNT(*) (C18) paths read through the manifest cache, LAZILY --- + // (fallback-to-SDK on a cache-read failure is not unit-tested: IcebergManifestCache is final so it cannot be + // made to throw, exactly as the pre-existing synchronous planFileScanTask fallback is untested; the streaming/ + // count catch(Exception)+recordFailure mirrors that path verbatim.) + + @Test + public void streamSplitsManifestCacheEnabledMatchesSdkPathAndConsumesCache() throws IOException { + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + table.newAppend() + .appendFile(dataFile(PartitionSpec.unpartitioned(), "/d/a.parquet", 100, null, null)) + .appendFile(dataFile(PartitionSpec.unpartitioned(), "/d/b.parquet", 200, null, null)) + .appendFile(dataFile(PartitionSpec.unpartitioned(), "/d/c.parquet", 300, null, null)) + .commit(); + IcebergTableHandle handle = new IcebergTableHandle("db1", "t1"); + + // Cache OFF: the streaming SDK planFiles() path (the pre-PERF-04 behavior). + List sdk = drain(new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table)) + .streamSplits(emptySession(), handle, Collections.emptyList(), Optional.empty(), -1L)); + // Cache ON: streaming must now read manifests THROUGH the cache and yield the SAME files (PERF-04 C17). + // MUTATION: streaming still bypasses the cache (scan.planFiles()) -> cache stays empty -> red. + IcebergManifestCache cache = new IcebergManifestCache(); + List cached = drain(manifestProvider(manifestCacheProps(), table, cache) + .streamSplits(emptySession(), handle, Collections.emptyList(), Optional.empty(), -1L)); + + Assertions.assertEquals(sortedPaths(sdk), sortedPaths(cached)); + Assertions.assertEquals(3, cached.size()); + Assertions.assertTrue(cache.size() > 0, "the streaming path must populate the manifest cache"); + } + + @Test + public void streamSplitsManifestCachePrunesPartitionLikeSdk() throws IOException { + PartitionSpec spec = PartitionSpec.builderFor(PART_SCHEMA).identity("p").build(); + Table table = createTable("pt", PART_SCHEMA, spec); + table.newAppend() + .appendFile(dataFile(spec, "/d/p1.parquet", 100, null, "p=1")) + .appendFile(dataFile(spec, "/d/p2.parquet", 100, null, "p=2")) + .commit(); + IcebergTableHandle handle = new IcebergTableHandle("db1", "pt"); + Optional wherePeq1 = Optional.of(eqInt("p", 1)); + + List sdk = drain(new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table)) + .streamSplits(emptySession(), handle, Collections.emptyList(), wherePeq1, -1L)); + IcebergManifestCache cache = new IcebergManifestCache(); + List cached = drain(manifestProvider(manifestCacheProps(), table, cache) + .streamSplits(emptySession(), handle, Collections.emptyList(), wherePeq1, -1L)); + + // Partition prune (ManifestEvaluator + residual) keeps only p=1 in BOTH paths. MUTATION: the lazy iterator + // dropping the residual/metrics prune -> p=2 leaks in -> sizes differ -> red. + Assertions.assertEquals(sortedPaths(sdk), sortedPaths(cached)); + Assertions.assertEquals(1, cached.size()); + Assertions.assertTrue(cached.get(0).getPath().get().endsWith("p1.parquet")); + } + + @Test + public void streamSplitsManifestCacheFlatMapsAcrossDataManifests() throws IOException { + // Three separate appends -> three data manifests. The lazy flat-map iterator must walk ALL of them (not + // just the first) and yield every file. MUTATION: advance() not advancing past the first manifest -> < 3. + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + table.newAppend().appendFile(dataFile(PartitionSpec.unpartitioned(), "/d/a.parquet", 100, null, null)) + .commit(); + table.newAppend().appendFile(dataFile(PartitionSpec.unpartitioned(), "/d/b.parquet", 200, null, null)) + .commit(); + table.newAppend().appendFile(dataFile(PartitionSpec.unpartitioned(), "/d/c.parquet", 300, null, null)) + .commit(); + Assertions.assertTrue(table.currentSnapshot().dataManifests(table.io()).size() >= 2, + "precondition: multiple data manifests"); + IcebergTableHandle handle = new IcebergTableHandle("db1", "t1"); + + IcebergManifestCache cache = new IcebergManifestCache(); + List cached = drain(manifestProvider(manifestCacheProps(), table, cache) + .streamSplits(emptySession(), handle, Collections.emptyList(), Optional.empty(), -1L)); + Assertions.assertEquals(3, cached.size(), "the lazy iterator must flat-map across all data manifests"); + } + + @Test + public void countPushdownManifestCacheMatchesCountAndReadsLazily() { + // Three appends -> three data manifests; record counts 10+20+30 = total-records 60 (snapshot summary). + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + table.newAppend().appendFile(dataFile(table.spec(), "s3://b/db/t1/f1.parquet", 1024, null, null)).commit(); + table.newAppend().appendFile(dataFile(table.spec(), "s3://b/db/t1/f2.parquet", 2048, null, null)).commit(); + table.newAppend().appendFile(dataFile(table.spec(), "s3://b/db/t1/f3.parquet", 3072, null, null)).commit(); + int totalManifests = table.currentSnapshot().dataManifests(table.io()).size(); + Assertions.assertTrue(totalManifests >= 2, "precondition: multiple data manifests"); + + List sdk = planCount( + new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table)), null, true); + IcebergManifestCache cache = new IcebergManifestCache(); + List cached = planCount(manifestProvider(manifestCacheProps(), table, cache), + emptySession(), true); + + // Same collapsed single range + same count (from the snapshot summary). The placeholder file path may + // differ (SDK planFiles' ParallelIterable order is non-deterministic), so assert count + shape, not path. + Assertions.assertEquals(1, sdk.size()); + Assertions.assertEquals(1, cached.size()); + Assertions.assertEquals(60L, cached.get(0).getPushDownRowCount()); + Assertions.assertEquals(sdk.get(0).getPushDownRowCount(), cached.get(0).getPushDownRowCount()); + // Lazy early stop: COUNT needs only the first surviving file, so it must NOT read every data manifest. + // MUTATION: routing count through the materialized cache path -> reads all manifests -> size == total -> red. + Assertions.assertTrue(cache.size() >= 1 && cache.size() < totalManifests, + "count reads lazily (stops at the first file's manifest), not the whole table"); + } + + @Test + public void countPushdownManifestCacheEmptyNullSnapshotReturnsNoRanges() { + // A never-appended table has no current snapshot; getCountFromSnapshot returns 0 (>=0) so planCountPushdown + // runs with a null-snapshot scan. cacheBackedFileScanTasks must keep the null-snapshot guard (empty + // iterable), not NPE. MUTATION: dropping the guard -> NPE on scan.snapshot() -> red. + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + IcebergManifestCache cache = new IcebergManifestCache(); + List cached = planCount(manifestProvider(manifestCacheProps(), table, cache), + emptySession(), true); + Assertions.assertTrue(cached.isEmpty(), "empty (null-snapshot) count with cache enabled yields no ranges"); + } + @Test public void planScanManifestCacheEmptyTableReturnsNoRanges() { Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); From 50ab72a290e5b61f98ea6b2af79cda59a3ce5681 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 09:32:59 +0800 Subject: [PATCH 11/47] [doc](catalog) fe-connector-iceberg PERF-04: design + summary + tracker updates; next = PERF-05 Records the lazy manifest-cache reconnect (PERF-04, commit 2e5f393779c): design doc (with the resolved legacy-parity conflict + red-team fold-ins), summary, tasklist check-off, progress log, and a fresh HANDOFF pointing at PERF-05 (the information_schema.tables per-table loadTable-for-comment, C9) with the fe-core loop constraint and the first-query N-load floor flagged. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 59 +++++++------- ...04-manifest-cache-lazy-reconnect-design.md | 76 +++++++++++++++++++ ...4-manifest-cache-lazy-reconnect-summary.md | 43 +++++++++++ plan-doc/perf-hotpath-iceberg/progress.md | 13 ++++ plan-doc/perf-hotpath-iceberg/tasklist.md | 10 ++- 5 files changed, 169 insertions(+), 32 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-04-manifest-cache-lazy-reconnect-design.md create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-04-manifest-cache-lazy-reconnect-summary.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index aabe24ba15d4c5..ea8618aa2efe47 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -6,52 +6,55 @@ --- -# ✅ 已完成（session 1~3） +# ✅ 已完成（PERF-01 ~ 04） -- **PERF-01**（commit `484f0e0c125`）：胖 handle（`IcebergTableHandle.transient resolvedTable`，查询内）+ 跨查询 `IcebergTableCache`（挂 `IcebergConnector`，**凭证 gate**）。同表规划期远端 `loadTable` 3~7→1。 -- **PERF-02**（commit `518d0599cbf`）：跨查询 `IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、值 raw 分区列表、**无凭证 gate**）。分区表分析期 PARTITIONS 扫描每查询→每快照一遍；MTMV 4~6 遍→1。 -- **PERF-03**（commit `0b96f2e6c78`）：跨查询 `IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、值格式名 `String`、**无凭证 gate**）。无 `write-format`/`write.format.default` 的表每查询整表格式推断 `planFiles()`→每快照一遍。只缓存推断兜底、失败不缓存、映射/抛点在 `getOrLoad` 之外；写路径 7 调用点未动（留 PERF-07）。小结 `designs/FIX-PERF-03-*-summary.md`。 -- 三轮均全 iceberg 模块单测绿（932→943→957 / 0 fail / 1 skip），checkstyle 绿，0 回归。 -- **可复用产物**：三套 `(TableIdentifier[,snapshotId])` + `MetaCacheEntry` 缓存基建（table/partition/format）；**凭证 gate 判据**（值含 FileIO/凭证→gate；纯元数据→无 gate，已三次印证）；「缓存只存 loader 原始输出、映射与抛点放 getOrLoad 之外、失败透传 unchecked 不入缓存」范式；`loadCountForTest` 度量守门法。 +- **PERF-01**（`484f0e0c125`）：胖 handle（查询内）+ 跨查询 `IcebergTableCache`（**凭证 gate**）。一次规划远端 loadTable 3~7→1。 +- **PERF-02**（`518d0599cbf`）：跨查询 `IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、**无 gate**）。分区表分析期 PARTITIONS 扫描每查询→每快照一遍。 +- **PERF-03**（`0b96f2e6c78`）：跨查询 `IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、值格式名、**无 gate**）。无 write-format 表整表格式推断每查询→每快照一遍。 +- **PERF-04**（`2e5f393779c`）：抽惰性 `cacheBackedFileScanTasks`（delete 索引 eager + data manifest 惰性扁平映射）三处复用，把 manifest cache 接回大表流式 + COUNT(*)（**缓存与不 OOM 兼得**，否决审计"退回物化"因重引 legacy OOM）。小结 `designs/FIX-PERF-04-*-summary.md`。 +- 四轮全 iceberg 模块单测绿（932→943→957→962 / 0 fail / 1 skip），checkstyle 绿，0 回归。 +- **可复用产物**：三套 `(TableIdentifier[,snapshotId])` + `MetaCacheEntry` 缓存基建（table/partition/format）；**凭证 gate 判据**（值含 FileIO/凭证→gate，纯元数据→无 gate，已三次印证）；「缓存只存 loader 原始输出、映射/抛点放 getOrLoad 之外、失败透传 unchecked 不入缓存」范式；`loadCountForTest`/`cache.size()` 度量守门法；「delete 索引 eager + 惰性扁平映射」范式。 --- -# 🆕 下一个 session = **PERF-04（IcebergManifestCache 两条旁路接回，C17 C18）** +# 🆕 下一个 session = **PERF-05（C9 information_schema.tables 每表 loadTable 只为取 comment）** -## ⚠️ 触发面很窄（P1）：仅当用户显式开 `meta.cache.iceberg.manifest.enable=true`（**默认 OFF**）。修的是"缓存开了却在两条路径上被绕过"。 +## 病灶（触发面广、教科书级"伪装成轻访问器"） + +`FrontendServiceImpl.listTableStatus` 的 **fe-core** `for (TableIf table : tables)` 循环里**无条件**调 `status.setComment(table.getComment())`（不看请求是否要 comment 列，且在 `table.readLock()` 下**串行**）→ `PluginDrivenExternalTable.getComment`（fe-core）→ `IcebergConnectorMetadata.getTableComment`（连接器）→ **每表一次 `loadTable`**（HMS getTable RPC + metadata.json GET，或 REST loadTable）。一个库 N 张 iceberg 表 = 一条 `SELECT * FROM information_schema.tables` / `SHOW TABLE STATUS` 付 **N 次串行远端 load**（每次 50~300ms），几百表数十秒~分钟，BI 工具高频触发。 ## 第一件事（立项流程见 README §单项立项流程） -1. **复核（动码前，行号信 grep 不信文档；PERF-01/02/03 后行号已漂移）**。两条独立旁路，同一病根（manifest cache 只接在同步 `planFileScanTask` 分支，`gate isManifestCacheEnabled`）： - - **C17（streaming 大表踢出 cache）**：`FileQueryScanNode.createScanRangeLocations isBatchMode → PluginDrivenScanNode.computeBatchMode（streamingSplitEstimate ≥ num_files_in_batch_mode → streamingBatch，无 cache gate）→ startStreamingSplit → IcebergScanPlanProvider.streamSplits → scan.planFiles()`（裸 planFiles，**不走 cache**）。**讽刺点**：文件数 ≥ `num_files_in_batch_mode`（默认 1024）才走 streaming，而这**恰好是 manifest cache 想加速的大表**；legacy 在 batch 模式仍走 cache。 - - **C18（COUNT(*) 下推绕过 cache）**：`getSplits planScan(countPushdown=true) → planScanInternal（getCountFromSnapshot≥0）→ planCountPushdown → scan.planFiles()`，在到达 cache 分支（`planFileScanTask`）**之前 return**；且 `ParallelIterable` 激进提交所有 manifest 读任务（虽只要计数聚合）。 -2. **设计**（写 `designs/FIX-PERF-04-...-design.md`）。审计建议方向： - - C17 = **cache 开启时 `streamingSplitEstimate` 返回 -1**（= 本模块"count 不可下推/未知"同款 sentinel），令 `computeBatchMode` 退回同步物化 `planFileScanTask` 路径（走 cache）。**⚠ 关键风险须论证**：streaming 是**大表 OOM 保护**（不持全量 task 列表）。强制大表走同步物化会否重新引入 OOM？—— 必须核 legacy「batch 模式如何同时用 cache 又不 OOM」（读 `git show 83585fd5097^:.../IcebergScanNode` 附近），确认"cache 开→退同步"是否真 legacy-parity、还是要更细（如仅当 cache 命中才退）。这条别当"一行修复"轻下。 - - C18 = count 分支改走 `planFileScanTask`（honors cache），而非裸 `scan.planFiles()`。较独立、风险小，可先做。 -3. **红队 + TDD**：度量守门 = manifest cache 开启时，streaming 大表 / COUNT(*) 下推的 manifest 读**命中 cache**（非每次裸 planFiles）。可仿现有 `IcebergManifestCache` 测试 + `getManifestCacheValue` 计数。 -4. 守铁律：改动放**连接器侧**（`IcebergScanPlanProvider` 的 `streamingSplitEstimate`/`planCountPushdown`）；C17 的 -1 sentinel 是连接器决策，别在 fe-core 加 source-specific 分支。 +1. **复核（动码前，行号信 grep 不信文档；PERF-01~04 后行号已漂移）**：grep `FrontendServiceImpl.listTableStatus`(fe-core) 的循环 + `setComment` 是否仍无条件、是否在 readLock 下；`PluginDrivenExternalTable.getComment` 是否有缓存字段；**`IcebergConnectorMetadata.getTableComment` 当前是裸 `loadTable` 还是已走 PERF-01 的 `resolveTableForRead`（tableCache）**？——这决定修法起点。 +2. **⚠ 铁律约束（关键）**：循环在 **fe-core**（`FrontendServiceImpl` / `PluginDrivenExternalTable.getComment`），**「fe-core 源只出不进」→ 不得改 fe-core 循环、不得给 fe-core 加缓存 comment 字段**。修法必须**连接器侧**。 +3. **设计**（写 `designs/FIX-PERF-05-...-design.md`），候选方向： + - **(a) 连接器侧 comment 缓存**（键 `TableIdentifier`、值 comment `String`、**无凭证 gate**——纯元数据）：`getTableComment` 先查缓存。**镜像 PERF-03 format cache**。**注意**：这消不掉**首次** N 次 load（无批量 SPI），但让**重复** information_schema 查询坍缩（BI 高频→除以 QPS-over-TTL）；REFRESH TABLE/DB/CATALOG 失效。 + - **(b) 若 getTableComment 尚未走 tableCache**：至少改走 PERF-01 的缓存 resolve，让"已被扫描加载过的表" + "重复查询"免费（与 (a) 可叠加）。 + - **批量 load（HMS `get_table_objects_by_name`）不可行**：需新 SPI 批量方法 + 改 fe-core 循环（fe-core 加法，越界）——排除，记为已知局限（首次 N load 无法在本约束下消除）。 +4. **红队 + TDD**：度量守门 = 重复取同表 comment 时远端 `loadTable`/`getTableComment` 次数从 N→命中（仿 `loadCountForTest`）；gate = 三目录（plain/vended/session）comment 缓存均建（无 gate）+ REFRESH 失效。 +5. 守铁律：缓存放连接器侧，值纯元数据无 gate；不碰 fe-core。 -## ⚠️ 关键认知（承 PERF-01/02/03） +## ⚠️ 关键认知 -- **缓存都挂 `IcebergConnector`**（长生命周期）；键统一 `(TableIdentifier[, snapshotId])`。`IcebergManifestCache` 是**已存在**的（path-keyed、no-TTL、cap 100k），PERF-04 不新建缓存，只是把两条绕过它的路径接回去。 -- **凭证 gate 判据**：值含 FileIO/凭证→gate；纯元数据→无 gate。（PERF-04 不涉新缓存值，无关。） -- **PERF-05（C9 information_schema comment 每表 loadTable）改动小、收益明确**，若 PERF-04 的 OOM 风险论证卡住，可临时插队 PERF-05 热身（README 允许按收益/风险微调顺序，但默认按 §5 优先级）。 -- **scan 级 format_type 决定 V1/V2 scanner**（全局 memory）：PERF-04 改 streaming/count 决策**不碰** format_type，但注意 streaming↔batch 模式切换别误动 `getFileFormatType` 路径。 +- **缓存都挂 `IcebergConnector`**（长生命周期）；comment 缓存键用 `TableIdentifier`（comment 是表级属性，与快照无关，故**不需要 snapshotId**——与 format/partition 缓存不同，那两个随快照变，comment 随 DDL 变靠 REFRESH 失效即可）。 +- **凭证 gate 判据**：comment 是纯元数据字符串（无 FileIO/凭证）→ **无 gate**（同 partition/format 缓存）。 +- **首次 N load 是本约束下的硬下限**：诚实写进设计 Risk（不假装消除），只优化重复。 --- # 🧰 构建/验证坑（**实证，务必照做**） -1. **可靠跑单测 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`**（绝对 `-f /fe/pom.xml`）。原因：本 worktree `${revision}` CI 版本 + 未 flatten 的已装 pom → `-pl iceberg` 单模块解析不到 `fe-connector:pom:${revision}`，必须 `-am`；且 `-am test` 只到 test 相不产 hms-hive-shade 的 shade jar（缺 `HiveConf`），故用 **`install`**。`-Dtest=`（=`ls src/test/.../*Test.java` 拼逗号，本轮 55 类）让上游 0 匹配测试快跳。**checkstyle 单独跑** `mvn checkstyle:check -pl fe-connector/fe-connector-iceberg`（测试跑时 `-Dcheckstyle.skip=true`）。 +1. **可靠跑单测 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`**（绝对 `-f /fe/pom.xml`）。原因：本 worktree `${revision}` + 未 flatten 已装 pom → `-pl iceberg` 解析不到 `fe-connector:pom:${revision}`，必须 `-am`；`-am test` 只到 test 相不产 hms-hive-shade shade jar（缺 `HiveConf`），故用 **`install`**。`-Dtest=`（=`ls src/test/.../*Test.java` 拼逗号，本轮 55 类，存 `scratchpad/iceberg-tests.txt`）让上游 0 匹配测试快跳。**checkstyle 单独** `mvn checkstyle:check -pl fe-connector/fe-connector-iceberg`。 2. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 静默跳过）。 -3. **别用 `nohup ... &` 套在 Bash `run_in_background` 里**（本轮踩坑）：外层会立即随 `echo` 退出、maven 变孤儿仍在跑 → "task completed exit 0" 是假信号。直接 `mvn ... > log 2>&1`（run_in_background）或用 `Monitor`/`until grep -qE 'BUILD SUCCESS|BUILD FAILURE' log` 守。读日志 `BUILD SUCCESS`/`Tests run:` 行，别信 exit code。 -4. **checkstyle**：LineLength(120) 对 **test 源已 suppress**；主源 ≤120；import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA`，组内字母序、组间空行（`org.apache.iceberg.catalog.X` 排在 `org.apache.iceberg.T*` 之后、`org.apache.iceberg.io.*` 之前）。 -5. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add 改动文件。 -6. **并发探测**：本目录 `find fe -newermt '-120 sec'` + `pgrep -a -f maven`。`/mnt/disk1/yy/git/doris` 是另一 worktree 的构建，不动本 `wt-catalog-spi` 文件，非冲突。 -7. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 +3. **`mvn -q` 成功时不打 `BUILD SUCCESS`/`Tests run`（INFO 被抑制）**——本轮踩坑。验证成功用 **surefire XML 聚合**（`target/surefire-reports/TEST-*.xml` 的 tests/failures/errors 属性求和），别只看 exit code / grep BUILD 行。或去掉 `-q`。 +4. **别 `nohup ... &` 套在 Bash `run_in_background` 里**——外层立即随 echo 退出、maven 变孤儿，"completed exit 0" 是假信号。直接 `mvn ... > log 2>&1`（run_in_background）。 +5. **checkstyle**：LineLength(120) 对 test 源 suppress；主源 ≤120；import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行。**新用类记得 import**（本轮 `java.util.Iterator` 漏 import 编译挂）。 +6. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。 +7. **并发探测**：本目录 `find fe -newermt '-120 sec'` + `pgrep -a -f maven`。`/mnt/disk1/yy/git/doris` 是另一 worktree，不冲突。 +8. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 --- # 🗂 开放问题 -- **PERF-04 C17 的 OOM 风险**（见上第一件事 step 2）：cache 开启时把大表从 streaming 退回同步物化，是否重新引入 OOM？须核 legacy batch 模式如何兼顾 cache 与不 OOM，再定"退同步"的精确条件。这是 PERF-04 设计的核心待答项。 +- **PERF-05 首次 N load 硬下限**：本约束（fe-core 不可改）下，comment 缓存只能优化重复查询，首次 `SELECT * FROM information_schema.tables` 仍付 N 次串行 load。批量 load 需改 fe-core → 越界。设计须诚实记为已知局限（对齐 PERF-02 "范围决定" 的诚实口径）。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-04-manifest-cache-lazy-reconnect-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-04-manifest-cache-lazy-reconnect-design.md new file mode 100644 index 00000000000000..de9405d4bbc0a9 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-04-manifest-cache-lazy-reconnect-design.md @@ -0,0 +1,76 @@ +# FIX-PERF-04 — Design（惰性 manifest-cache 接回流式规划 + COUNT(*) 下推） + +> 覆盖审计发现 **C17（流式路径绕过 manifest cache）+ C18（COUNT(*) 下推绕过 manifest cache）**。 +> 用户 2026-07-18 拍板方向 = **惰性+缓存兼得**（把缓存规划改成边读边吐，两路径接回）。基线 HEAD `584ef3822d8`。行号信 grep。 + +## Problem + +manifest cache（`meta.cache.iceberg.manifest.enable`，**默认 OFF**，opt-in）把解析过的 manifest 缓在 FE 内存里，让重复查询把远端 manifest 读变内存命中。**只接在同步物化路径** `planFileScanTask`（gate `isManifestCacheEnabled`）→ `planFileScanTaskWithManifestCache`。两条路径即使开了缓存也绕过它： + +- **C17（主）**：文件数 ≥ `num_files_in_batch_mode`（默认 1024）的大表走**流式**规划（`streamSplits → scan.planFiles()`，SDK 裸扫、不物化整表任务列表 = OOM 保护），**不读缓存**。讽刺：大表恰是缓存最想加速的对象，命中率 0。 +- **C18（次）**：`COUNT(*)` 下推 `planCountPushdown → scan.planFiles()`（只取第一个数据文件拼占位切片；行数其实来自 `getCountFromSnapshot` 快照摘要），也**不读缓存**，且 `ParallelIterable` 激进提交所有 manifest 读任务（虽只要第一个）。 + +## 复核确认的冲突（已解，见 progress） + +审计说"legacy batch 模式走缓存"——**核 legacy `IcebergScanNode`（`6fef25709d3^`）确认，但补全另一半**：legacy `doStartSplit → planFileScanTask`（610）**确走缓存**；**但 legacy 走缓存时是一次性物化整表任务列表的**（`planFileScanTaskWithManifestCache` 建 `List tasks`），即 **legacy 开缓存的大表根本没 OOM 保护**（OOM 保护只在缓存关时的 `splitFiles` 惰性分支）。迁移故意让大表一律流式（OOM 安全）换掉了缓存收益。故审计的"缓存开就退回物化"（= 恢复 legacy）会**重引 OOM 风险**——已否决。 + +## Root Cause + +缓存规划（`planFileScanTaskWithManifestCache`）**物化**整表任务列表（phase 2 攒 `List`），与流式的惰性 OOM 保护互斥。故大表要么走缓存（物化/OOM）要么走流式（惰性/无缓存）——二选一。**解法 = 让缓存规划本身惰性化**（边读 manifest 边吐任务、不攒全表），流式与 COUNT 都复用它 → 缓存与不 OOM 兼得。 + +## Design（连接器侧，零 fe-core 改动；核心 = 抽一个惰性缓存迭代器，三处复用） + +### 1. 新建 `cacheBackedFileScanTasks(scan, session, table, filter, String statsQueryId)` → `CloseableIterable`（惰性） +从现 `planFileScanTaskWithManifestCache` 抽出，产出**整文件**（未切片）`BaseFileScanTask`： +- **保留 `snapshot==null` 守卫**（现 1751-1754）：空表返回空 `CloseableIterable`（空表 COUNT 会带 null 快照走到这里，丢守卫会 NPE）。 +- **Phase 1（eager，有界）**：读所有匹配 **DELETE** manifest（经 cache）建 `DeleteFileIndex`。与现状/legacy/iceberg `planFiles` 一致——iceberg 的 `planFiles` 本就先读全部 delete manifest 建索引再吐 data 任务，故 eager delete 不比现流式差（现流式 `scan.planFiles()` 也 eager 读 delete）。delete manifest 通常远少于 data，且这是产任一 data 任务的正确性前提。 +- **Phase 2（LAZY）**：返回一个 `CloseableIterable`，其迭代器**扁平映射**匹配 **DATA** manifest（`getMatchingManifest`），逐个经 cache 读、逐文件 `InclusiveMetricsEvaluator` + `ResidualEvaluator` 剪枝、`DeleteFileIndex.forDataFile` 挂 delete，产出存活 `BaseFileScanTask`——**不攒 list**。峰值内存 = delete 索引 + 当前一个 manifest 的文件 + 切片队列背压，**非整表**。 +- **stats 线程安全（红队 MED）**：manifest cache 的命中/未命中计数器 `getManifestCacheValue(m, table, queryId)` 是**按 queryId 单线程假设**的（无锁 `stats.hits++`）。流式的 Phase 2 在引擎 pump 线程跑、Phase 1 在调用线程跑 → 两线程同 queryId 竞争。故 `statsQueryId` **可空**：流式传 **null** → 用**无 stats** 重载 `getManifestCacheValue(m, table)`（流式今天本就不报 cache stats，行为不变、消竞争）；COUNT/同步单线程 → 传 `session.getQueryId()` 用 stats 重载（不变）。 + +迭代器 `ManifestCacheFileScanTaskIterator`（内部类，`CloseableIterator`）：持 `CloseableIterator manifestIt` + 当前 `Iterator` + 当前 spec/residual + `deleteIndex`/evaluators + 缓冲 `next`；`advance()` 拉取下一个存活任务（必要时跨 manifest）；`close()` 关 `manifestIt`。三个消费方均须在 try-with-resources / SplitPlan / streaming source.close 内关闭它（单次关，`withNoopClose` 防同步路径二次关）。 + +### 2. 重构 `planFileScanTaskWithManifestCache`（同步物化路径，小表 < 阈值） +改成：物化 `cacheBackedFileScanTasks(...)` 成 `List` → `determineTargetFileSplitSize(list)`（**启发式**切片大小，小表要它给 BE 并行度）→ `SplitPlan(splitFiles(withNoopClose(list), size), size)`。**字节等价**（同任务、同大小）——现有同步 cache parity 测试必须继续绿（守门）。 + +### 3. `streamSplits`（C17） +`isManifestCacheEnabled()` 时：`whole = cacheBackedFileScanTasks(scan, session, table, filter, null)`（**null** = 无 stats，消 pump/调用线程竞争）；`tasks = TableScanUtil.splitFiles(whole, sliceSize)`（**固定** `file_split_size`/`max_file_split_size`，与现流式同款——大表本就固定大小，**切片大小不变**）→ 喂现 `IcebergStreamingSplitSource`。**唯一变化 = 大表现在命中/填充缓存，切片大小/流式/OOM 边界全不变**。 +- **失败回退**：调 `cacheBackedFileScanTasks(...)`（含 eager phase-1）包 **`try/catch(Exception)`**（红队 HIGH：`IcebergManifestCache.loadManifestCacheValue` 把 IOException 重抛成 `RuntimeException`，`deleteManifests` 抛 `RuntimeIOException`，故须 `Exception` 非 `IOException`，镜像同步 `planFileScanTask:1731` 的 `catch (Exception)`）→ `manifestCache.recordFailure(queryId)` + 退回 `scan.planFiles()`（SDK 流式）。phase-2 惰性失败经 streaming source 的 `hasNext` 路由到引擎 setException（与现 SDK 流式失败同路径；且经 cache 读 manifest 失败 ⇔ SDK 读同一 manifest 也失败，无"cache 挂但 SDK 成"场景）。 +- 缓存关：现 `scan.planFiles()` 不变。 + +### 4. `planCountPushdown`（C18） +`isManifestCacheEnabled()` 时：迭代 `cacheBackedFileScanTasks(scan, session, table, filter, session.getQueryId())`（COUNT 单线程 → 用 stats 重载）取**第一个**任务 buildRange（惰性 → 只读到出第一个文件的 manifest 即停，避免 `ParallelIterable` 全提交）。包 **`try/catch(Exception)`** → 退回 `scan.planFiles()`。需把 `filter` 从 `planScanInternal` 透传给 `planCountPushdown`（现未传；count 下推通常无 WHERE，filter 多为 alwaysTrue，透传保通用+正确）。行数仍来自 `realCount`（快照摘要），BE 不读文件——**首文件取自剪枝后存活文件，与 SDK 路径 count 值相同**（占位切片的 file path 因 SDK `ParallelIterable` 乱序而 OFF/ON 不同，属正常，测试勿断言 path 相等）。 +- **scan-metrics**：cache 路径不调 `scan.planFiles()` → 不触发 `IcebergScanProfileReporter`；与**现有同步 cache 路径一致**（它也不调 planFiles），仅 count+cache 少一份 scan profile，可接受。 +- 缓存关：现 `scan.planFiles()` 不变。 + +### 5. `streamingSplitEstimate` — **不改** +批量/流式**决策**（按 manifest 元数据计数，cheap）不变；只改流式**执行**读法。OOM 边界（阈值决策）原封不动。 + +## Parity 论证 + +- **同任务**：三路径都源自 `cacheBackedFileScanTasks`，其 phase-1/phase-2 剪枝逻辑 = 现 `planFileScanTaskWithManifestCache` 逐字搬（只是惰性化）→ 产出与现同步 cache 路径**逐任务相同**；现同步 cache 路径已声明与 `scan.planFiles()` legacy-parity。故惰性化**不引入新 parity 风险**（不发明新规划，只把既有缓存规划改惰性）。 +- **切片大小**：小表同步 = 启发式（不变）；大表流式 = 固定（不变）；COUNT = 单切片无关。 +- **失败语义**：eager phase-1 失败可回退 SDK（superset of 现流式，且 ≥ 现同步 cache 的回退）；phase-2 失败路由引擎（同现流式）。 +- **delete 索引 eager**：= 现流式 `scan.planFiles()` 与现同步 cache 与 legacy，无回退。 + +## Implementation Plan（连接器侧，逐处） + +1. `IcebergScanPlanProvider`：抽 `cacheBackedFileScanTasks(...)` + 内部类 `ManifestCacheFileScanTaskIterator`；`planFileScanTaskWithManifestCache` 改为物化它；`streamSplits`/`planCountPushdown` 加 cache 分支 + 回退；`planScanInternal` 给 `planCountPushdown` 透传 `filter`。 +2. 无新缓存、无新字段、无 fe-core/thrift/BE 改动。 + +## Test Plan + +- **流式 cache parity（新）** `IcebergScanPlanProviderTest`（仿 `planScanManifestCacheEnabledMatchesSdkPathAndConsumesCache`）：真表，`streamSplits` 缓存 OFF（SDK）vs ON（惰性 cache）→ 排序路径相等 + `cache.size()>0`（缓存被填/命中）。分区表 + WHERE 剪枝一例。**MUTATION**：流式仍走 SDK → cache 空 → 红。 +- **COUNT cache parity（新）**：`planScan(countPushdown=true)` 缓存 OFF vs ON → **同 count（table_level_row_count）+ 均为单个合法 range（其 path 是表内某文件）**；**不断言 path 相等**（红队 MED：SDK `planFiles` 的 `ParallelIterable` 乱序令首文件 OFF/ON 不定）。+ `cache.size()>0`；惰性早停（多 manifest 表断言 `cache.size() < 总 data manifest 数`）。 +- **空表 COUNT + cache（新，红队 MED）**：空表（null 快照）`planScan(countPushdown=true)` 缓存 ON → 0 range（守卫保住、不 NPE）。 +- **惰性迭代器行为（新）**：drain 全量、close-before-iterate 不抛、exhausted `next()` 抛（镜像现 `streamSplits*` 测试）；空表/全剪空 → 0 任务。 +- **回退（新）**：注入 phase-1 IOException（可用抛异常的 fake manifestCache 或坏 manifest）→ 流式/count 退回 SDK 且 `recordFailure` 记账。 +- **同步 cache 路径不回归**：现 `planScanManifestCache*` 全绿（物化路径字节等价守门）。 +- 全 iceberg 模块 UT 绿 + checkstyle 绿。 + +## Risk + +- **中**：共用热路径（流式 + COUNT + 同步 cache 三者现共源一个惰性方法）。缓解：同步路径物化同一 iterable → 现同步 cache parity 测试守字节等价；新增流式/COUNT cache-parity（vs SDK）守新路径；惰性迭代器行为测试守 close/exhaust。 +- 惰性迭代器手写（扁平映射 + close 语义），有实现坑（缓冲/推进/关闭）→ 测试覆盖。 +- delete 索引 eager 内存 = 现状，未新增；如超大 MOR 表 delete 多则同现状受影响（非本任务回归）。 +- **缓存填充变化（红队 LOW，本就是目的）**：大表流式现在会**填充** manifest cache（每读一个 data manifest 入缓存），改变 eviction 动态、增 FE 堆压力、可能挤掉小表条目。容量有界（100000）不会 OOM；这正是"让大表享受缓存"的题中之义，用户已 opt-in。 +- 触发面窄（opt-in 缓存默认关）→ 影响面可控。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-04-manifest-cache-lazy-reconnect-summary.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-04-manifest-cache-lazy-reconnect-summary.md new file mode 100644 index 00000000000000..2292fb8f384315 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-04-manifest-cache-lazy-reconnect-summary.md @@ -0,0 +1,43 @@ +# FIX-PERF-04 — Summary（惰性 manifest-cache 接回流式 + COUNT(*)） + +> 权威设计见 [`FIX-PERF-04-manifest-cache-lazy-reconnect-design.md`](./FIX-PERF-04-manifest-cache-lazy-reconnect-design.md)。本文件只记落地结果。 +> commit `2e5f393779c`（`[perf]`，iceberg 自包含）。基线 HEAD `584ef3822d8`。用户 2026-07-18 拍板方向 = 惰性+缓存兼得。 + +## Problem + +`IcebergManifestCache`（`meta.cache.iceberg.manifest.enable`，**默认 OFF**，opt-in）只接在同步物化路径。两条路径即使开缓存也绕过它： +- **C17**：≥`num_files_in_batch_mode` 的大表走流式 `streamSplits → scan.planFiles()`（SDK 裸扫），而大表恰是缓存目标 → 命中率 0。 +- **C18**：`COUNT(*)` 下推 `planCountPushdown → scan.planFiles()` 只为取一个占位文件（行数其实来自快照摘要），也不走缓存且 `ParallelIterable` 全提交。 + +## 复核（推翻审计"一行修复"） + +核 legacy（`6fef25709d3^`）：batch 模式**确走缓存，但走缓存时一次性物化整表任务列表**（无 OOM 保护；OOM 保护只在缓存关时的惰性分支）。迁移用"一律流式（OOM 安全、无缓存）"换掉了它。审计建议的"缓存开→退回物化"会重引 legacy 就有的 OOM 风险 → 否决。 + +## Fix（连接器侧，零 fe-core 改动） + +抽 `cacheBackedFileScanTasks()`——一个**惰性** `CloseableIterable`，三处复用： +- **Phase 1（eager）** 读全部 delete manifest（经 cache）建 `DeleteFileIndex`（产任一 data 任务的前提，且与 SDK `planFiles`/legacy 同样 eager，delete manifest 少）； +- **Phase 2（LAZY）** 扁平映射 data manifest，逐个经 cache 读、逐文件剪枝、产整文件 `BaseFileScanTask`，**不攒整表**（峰值 = delete 索引 + 一个 manifest 的文件 + 切片队列）。 +- `planFileScanTaskWithManifestCache` 改为**物化**它（字节等价；小表保启发式切片大小）；`streamSplits` 缓存开时喂它给现流式 source（固定切片大小不变，大表现在命中缓存且仍 OOM 安全）；`planCountPushdown` 缓存开时迭代它**取第一个**（惰性早停，无 `ParallelIterable` 全提交）。 +- **决策不改**（`streamingSplitEstimate`）。**失败回退** `catch (Exception)`（缓存把失败重抛成 `RuntimeException`）→ `recordFailure` + 退 SDK。**`statsQueryId` 可空**：流式传 null（无 stats 重载，消 pump/调用线程对同 queryId 计数器的竞争）；同步/COUNT 单线程用 stats。 + +## 红队（7 攻击核心 sound，采纳修正） + +- **HIGH**：回退须 `catch (Exception)` 非 `IOException`（缓存重抛 RuntimeException）——已修。 +- **MED**：流式两阶段跨线程 → 流式改用无 stats 重载消竞争——已修。 +- **MED**：COUNT 对比测试不断言 path 相等（SDK `ParallelIterable` 乱序），改断言 count + 单切片——已采纳。 +- **MED**：保留 `snapshot==null` 守卫 + 空表 COUNT 用例——已采纳。 +- **LOW**：大表流式现在填充缓存（本就是目的、容量有界不 OOM）；COUNT+cache 少一份 scan profile（与现有 cache 路径一致）——已记录。 + +## Tests（`IcebergScanPlanProviderTest` +5） + +- 流式 cache parity vs SDK + cache consumed；分区剪枝 parity；**跨多 data manifest 扁平映射**（3 append → 3 range）；COUNT cache count-parity（60）+ **惰性早停**（`cache.size() < 总 manifest 数`）；空表 null 快照 COUNT → 0 range（守卫不 NPE）。 +- 同步 cache 路径既有测试继续绿（物化字节等价守门）。 +- **回退路径未单测**：`IcebergManifestCache` 是 `final` 无法造抛异常 seam，与既有同步 `planFileScanTask` 回退同样未单测；流式/COUNT 的 `catch(Exception)+recordFailure` 逐字镜像它。 + +## Result + +- 全 iceberg 模块 **962 pass / 0 fail / 0 error / 1 skip**（`install -am`），checkstyle 0 违规。 +- 减负：开缓存的大表流式规划从"每查询裸读全部 manifest（0 命中）"→"命中缓存且惰性不 OOM"；`COUNT(*)` 从"全提交 manifest 读"→"命中缓存 + 首文件即停"。 +- 修坑：`java.util.Iterator` 漏 import（编译）+ 同步空表路径 null session 下 `session.getQueryId()` NPE（改 null-safe）——均 CI 前本地捕获。 +- 可复用：一个 `(delete 索引 eager + data manifest 惰性扁平映射)` 缓存迭代范式，供后续同类"物化 vs 惰性"权衡参考。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index 6a37a042c55c13..3bf59aead1e4e5 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -58,3 +58,16 @@ - **结果**：全 iceberg 模块 **957 pass / 0 fail / 1 skip**（`install -am`），checkstyle 绿，0 回归（无需改任何现有测试）。summary 见 `designs/FIX-PERF-03-format-inference-cache-summary.md`。 - **踩坑/印证**：InMemoryCatalog `newAppend().commit()` 后同一 table 对象 `currentSnapshot()` 会刷新（度量守门测试 ORC+loadCount==1 通过即证），无需像 `dayPartitionedTable` 那样 reload；但 reload 仍是更稳的既有范式。 - **下一步**：见 HANDOFF —— PERF-04（IcebergManifestCache 两条旁路接回：C17 streaming 大表踢出 cache + C18 COUNT(*) 下推绕过 cache）。 + +--- + +## 2026-07-18 — session 3（续）：PERF-04 实现 + 全绿（commit `2e5f393779c`） + +- **复核 + 冲突求证（关键）**：C17/C18 = manifest cache（opt-in 默认关）被大表流式 + COUNT(*) 下推两条路径绕过。**审计说"legacy batch 走 cache 是回退"，我核 legacy `6fef25709d3^` 确认但补全另一半**：legacy batch 走 cache 时**一次性物化整表任务列表**（无 OOM 保护，OOM 保护只在缓存关时的惰性分支）；迁移用"一律流式（OOM 安全、无缓存）"换掉了缓存收益。**审计建议的"缓存开→退回物化"会重引 legacy 就有的 OOM 风险**——否决"一行修复"。 +- **决策上交用户（AskUserQuestion，三选项 A 恢复 legacy/B 维持现状/C 惰性+缓存兼得）**：用户选 **C**。C 可行性由代码证实：现 `planFileScanTaskWithManifestCache` 只在 Phase 2 攒 `List`，delete 索引(Phase 1)本就 eager（= SDK planFiles 也 eager 读 delete）→ 把 Phase 2 改惰性即得"缓存 + 不 OOM"。 +- **设计红队（独立 agent 对抗，7 攻击）**：核心思路(惰性迭代器三处复用)判 sound；抓 **1 HIGH**（回退 `catch` 须 `Exception` 非 `IOException`——缓存把失败重抛成 RuntimeException）+ 3 MED（跨线程 stats 竞争→流式用无 stats 重载；COUNT 对比测试勿断言 path 相等因 `ParallelIterable` 乱序；保 `snapshot==null` 守卫 + 空表 COUNT 用例）+ 2 LOW（大表流式现在填充缓存=目的；COUNT+cache 少一份 profile=与现有 cache 路径一致）。全并入。 +- **实现（连接器侧一个 `[perf]` commit）**：抽 `cacheBackedFileScanTasks()`（惰性 CloseableIterable）+ `manifestCacheGet`（stats/无 stats 重载分派）+ 内部类 `ManifestCacheFileScanTaskIterator`（扁平映射迭代器，schemaJson/specJson 提为循环不变量）；`planFileScanTaskWithManifestCache` 改物化它；`streamSplits`→`streamingFileScanTasks`（cache 开惰性+回退）；`planCountPushdown`→`countPushdownFileScanTasks`（cache 开取首文件+回退）+ 从 `planScanInternal` 透传 `filter`/`session`。 +- **测试（`IcebergScanPlanProviderTest` +5）**：流式 cache parity vs SDK + 命中；分区剪枝 parity；跨多 data manifest 扁平映射；COUNT count-parity(60) + 惰性早停(`cache.size()<总 manifest`)；空表 null 快照 COUNT 守卫。回退路径未单测（`IcebergManifestCache` final 无 seam，与既有同步回退同样未测，逐字镜像）。 +- **踩坑（CI 前本地捕获）**：① `java.util.Iterator` 漏 import（编译挂）；② 重构后同步 `planFileScanTaskWithManifestCache` 顶部无条件 `session.getQueryId()`，而既有空表用例传 **null session**（旧码 snapshot==null 早返回、从不碰 session）→ NPE；改 `statsQueryId = session != null ? getQueryId() : null`。 +- **结果**：全 iceberg 模块 **962 pass / 0 fail / 0 error / 1 skip**，checkstyle 0 违规，0 回归（现有测试全绿）。summary 见 `designs/FIX-PERF-04-*-summary.md`。 +- **下一步**：见 HANDOFF —— PERF-05（C9 information_schema.tables 每表 loadTable 只为取 comment）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index 9da361b793c947..3f90f9c03954f7 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -18,7 +18,7 @@ | PERF-01 | P0 | C1 C4 C6 C10 C16 | 一次规划 3~7 次远程 loadTable → 胖 handle(查询内单实例)+ 跨查询 IcebergTableCache(挂 Connector);~~convertPredicate 收窄~~已删(红队证伪) | — | ✅ 完成 | `484f0e0c125` | | PERF-02 | P0 | C7 C22 C23 | 分区视图每查询重扫 PARTITIONS 元数据表 → `(table,snapshotId)` 缓存(连接器侧,无凭证 gate);MTMV refresh pin 判定为多余(靠 latestSnapshotCache 稳定快照坍缩,不改 fe-core) | 与 01 共享快照 pin 机制 | ✅ 完成 | `518d0599cbf` | | PERF-03 | P0 | C2 C11 | #64134 复活：`file_format_type` 兜底走整表 planFiles → 跨查询 `(table,snapshotId)` memoize(连接器侧,无 gate);~~从枚举反推~~已否决(getFileFormatType 早于 planScan + 无过滤 vs 带谓词破 parity) | 与 01/02 共享快照 pin | ✅ 完成 | `0b96f2e6c78` | -| PERF-04 | P1 | C17 C18 | streaming / COUNT(*) 下推旁路 IcebergManifestCache → 两条旁路接回 | — | ⏳ | | +| PERF-04 | P1 | C17 C18 | streaming / COUNT(*) 下推旁路 IcebergManifestCache → 抽惰性 `cacheBackedFileScanTasks`(delete 索引 eager + data manifest 惰性扁平映射)三处复用;缓存与不 OOM 兼得(否决审计"退回物化"因重引 OOM) | — | ✅ 完成 | `2e5f393779c` | | PERF-05 | P1 | C9 | information_schema.tables 循环内每表 loadTable（只为拿 comment） → 随表缓存 / 惰性取 | — | ⏳ | | | PERF-06 | P1 | C3 | REST vended-cred 每 data/delete file 重建 StorageProperties+Configuration → scan 级 memo | — | ⏳ | | | PERF-07 | P1 | C20 | 一条 DML 3~5 次 load 同表 → 语句级 resolve 一次传递 | — | ⏳ | | @@ -53,9 +53,11 @@ ## P1 — 局部旁路/局部 hoist，触发面较窄或特定场景 -### [ ] PERF-04 — 簇4：IcebergManifestCache 两条旁路接回（C17 C18） -- **病灶**：manifest cache（`meta.cache.iceberg.manifest.enable`，默认 off）只接在同步路径 `planFileScanTask:1681`（gate `isManifestCacheEnabled:1832`）。**C17**：文件数 ≥ `num_files_in_batch_mode`（默认 1024）走 streaming `streamSplits:449 → scan.planFiles():463`，**恰好把 cache 瞄准的大表踢出缓存**（legacy batch 模式仍走 cache）。**C18**：COUNT(*) 下推 `planScanInternal:594-598 → planCountPushdown:1021 → scan.planFiles():1024` 在到 cache 分支前 return（`ParallelIterable` 还激进提交所有 manifest 读任务，虽只要第一个）。 -- **修复方向**：cache 开启时 streaming 判定返回 -1 退回同步物化路径（**一行 legacy-parity 修复** = C17）；count 分支改走 `planFileScanTask`（C18）。 +### [x] PERF-04 — 簇4：IcebergManifestCache 两条旁路接回（C17 C18） · ✅ `2e5f393779c` +- **病灶**：manifest cache（`meta.cache.iceberg.manifest.enable`，默认 off）只接在同步物化路径。**C17**：≥`num_files_in_batch_mode` 的大表走 streaming `streamSplits → scan.planFiles()`（SDK 裸扫，恰把 cache 目标大表踢出）。**C18**：COUNT(*) 下推 `planCountPushdown → scan.planFiles()` 只为取一个占位文件（行数来自快照摘要），也不走 cache 且 `ParallelIterable` 全提交。 +- **复核否决审计"一行修复"**：核 legacy(`6fef25709d3^`) batch 模式确走 cache **但一次性物化整表**（无 OOM 保护）；"退回物化"会重引 legacy 就有的 OOM 风险。用户拍板方向 = **惰性+缓存兼得**。 +- **落地**：抽惰性 `cacheBackedFileScanTasks`（delete 索引 eager + data manifest 惰性扁平映射，产整文件任务不攒整表），三处复用（同步物化它保启发式切片；流式喂它保 OOM 安全+固定切片+现在命中缓存；COUNT 迭代取首文件惰性早停）。决策不改；失败 `catch(Exception)` 退 SDK；`statsQueryId` 可空(流式 null 消跨线程 stats 竞争)。红队 7 攻击核心 sound，采纳 catch 类型/线程/测试 3 修正。 +- **收益**：开缓存大表流式从 0 命中→命中且不 OOM；COUNT 从全提交→首文件即停。全模块 962 UT 绿。 ### [ ] PERF-05 — C9：information_schema.tables 每表 loadTable - **病灶**：`FrontendServiceImpl.listTableStatus:719 for(table)` → `:755 setComment(table.getComment())`（**无条件**，不看请求是否要 comment 列）→ `PluginDrivenExternalTable.getComment:944 → IcebergConnectorMetadata.getTableComment:305 → loadTable`。N 表 = N 次串行远程 load；几百表的库一条 `SELECT * FROM information_schema.tables` 数十秒~分钟，BI 工具高频触发。教科书级"伪装成轻访问器"。 From fcdebb6075931180200724a496ed1a7832c1d08a Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 10:38:37 +0800 Subject: [PATCH 12/47] [perf](catalog) fe-connector-iceberg: cache the table comment for vended-credentials catalogs (PERF-05) Root cause: information_schema.tables / SHOW TABLE STATUS loops per-table in fe-core (FrontendServiceImpl.listTableStatus) and unconditionally calls table.getComment() -> PluginDrivenExternalTable.getComment -> IcebergConnectorMetadata.getTableComment -> loadTable(...), a full remote table load just to read the 'comment' property. N tables => N serial remote loads for one query, on a path BI tools hit constantly. Re-scope (the audit predates PERF-01): getTableComment now resolves through loadTable -> resolveTableForRead -> IcebergTableCache (PERF-01), so on a PLAIN catalog repeat information_schema queries are already cache hits. The remaining gap is credential-gated catalogs, where tableCache is null: they reload every table every query. This fix targets exactly that gap. Fix (connector-side, no fe-core change): add an ungated IcebergCommentCache (TableIdentifier -> comment String, mirrors IcebergTableCache), built ON THE CONNECTOR only for a REST vended-credentials catalog that is NOT session=user. getTableComment consults it when present; otherwise resolves directly (plain catalogs keep using tableCache; the comment cache is null there, no redundancy). A thrown load (view handle NoSuchTableException) is not cached and propagates to the caller's catch (still ""), so behavior is unchanged. REFRESH TABLE/DB/CATALOG invalidate it. Security scoping (this is why it is NOT gated on "tableCache == null"): session=user is DELIBERATELY excluded. There the loadTable call itself carries the querying user's per-request authorization; a connector-shared comment cache would serve one user's comment to another whose delegated token was never validated for that table -- a metadata disclosure and a regression vs today's always-live per-user path. Vended-credentials uses a single static catalog identity (every user loads the same table), so sharing a comment is safe. Honest limits: the FIRST query on any catalog still pays N loads (a batch load would need a new SPI + the fe-core loop change, out of scope); view comments stay uncached (their load throws); vended catalogs gain a comment-staleness window up to TTL (REFRESH-invalidated), the standard cache trade-off. Tests: IcebergCommentCacheTest (TTL stability, ttl<=0 disable, invalidate/Db/All, not-cached-on-failure); a metadata metric gate (repeated getTableComment -> one remote loadTable, loadCountForTest==1); connector gate asserting the cache is built ONLY for vended-non-session (null for plain, session=user, and vended+session=user) plus REFRESH invalidation. Full iceberg module UT: 973 run / 0 fail / 0 error / 1 skip; checkstyle 0 violations. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../iceberg/IcebergCommentCache.java | 111 +++++++++++ .../connector/iceberg/IcebergConnector.java | 33 +++- .../iceberg/IcebergConnectorMetadata.java | 26 +++ .../iceberg/IcebergCommentCacheTest.java | 177 ++++++++++++++++++ .../iceberg/IcebergConnectorCacheTest.java | 59 ++++++ .../iceberg/IcebergConnectorMetadataTest.java | 28 +++ 6 files changed, 432 insertions(+), 2 deletions(-) create mode 100644 fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCommentCacheTest.java diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java new file mode 100644 index 00000000000000..aa179082d3d234 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCommentCache.java @@ -0,0 +1,111 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.cache.CacheSpec; +import org.apache.doris.connector.cache.MetaCacheEntry; + +import org.apache.iceberg.catalog.Namespace; +import org.apache.iceberg.catalog.TableIdentifier; + +import java.util.concurrent.ForkJoinPool; +import java.util.function.Supplier; + +/** + * Per-catalog cache of an iceberg table's {@code comment} property (PERF-05), keyed by {@link TableIdentifier}. + * Kills the per-table remote {@code loadTable} that {@code information_schema.tables} / {@code SHOW TABLE STATUS} + * pays for EVERY table just to read its comment ({@code FrontendServiceImpl.listTableStatus} loops per table and + * unconditionally calls {@code getComment} -> {@link IcebergConnectorMetadata#getTableComment} -> + * {@code loadTable}). + * + *

Scope: REST vended-credentials catalogs ONLY (see {@link IcebergConnector}). Plain catalogs already + * reuse the raw {@link IcebergTableCache} (PERF-01) for the comment path, so this cache is not built for them (no + * redundancy). A {@code iceberg.rest.session=user} catalog is DELIBERATELY excluded: there the {@code loadTable} + * call itself carries the querying user's per-request authorization, and a connector-shared comment cache would + * serve one user's comment to another whose delegated token was never validated for that table (a metadata + * disclosure). Vended-credentials uses a single static catalog identity, so every user loads the same table and + * sharing a comment is safe. + * + *

No credential gate (unlike {@link IcebergTableCache}, like {@link IcebergPartitionCache} / + * {@link IcebergFormatCache}): the cached value is a bare comment {@link String} with no {@code FileIO} / + * credential. A comment changes only via external DDL, picked up through the REFRESH invalidate hooks. TTL is + * {@code meta.cache.iceberg.table.ttl-second}; {@code <= 0} disables (read live), so a no-cache catalog serves a + * fresh comment even when this object is built. Backed identically to {@link IcebergTableCache}: a contextual, + * access-TTL {@link MetaCacheEntry} with manual miss-load, so the remote load runs OUTSIDE Caffeine's compute + * lock and its exception (e.g. the view-handle {@code NoSuchTableException}) propagates verbatim and a failed + * load is not cached. Lives on the long-lived per-catalog {@link IcebergConnector}; a REFRESH CATALOG rebuilds it. + */ +final class IcebergCommentCache { + + private final MetaCacheEntry entry; + + IcebergCommentCache(long ttlSeconds, int maxSize) { + // Mirror IcebergTableCache: translate the connector's "<= 0 disables" contract to CacheSpec's ttl == 0 + // (disabled) rather than passing a negative value through, which CacheSpec reads as ttl == -1 "no + // expiration (enabled)". This ttl<=0 -> disabled mapping is load-bearing: a vended no-cache catalog builds + // this object but must NOT cache comments (operator "no meta cache" intent). + CacheSpec spec = ttlSeconds > 0 + ? CacheSpec.of(true, ttlSeconds, maxSize) + : CacheSpec.of(true, CacheSpec.CACHE_TTL_DISABLE_CACHE, maxSize); + this.entry = new MetaCacheEntry<>("iceberg-comment", null, spec, + ForkJoinPool.commonPool(), false, true, 0L, true); + } + + /** Caching is on only when the TTL is positive; ttl-second <= 0 means "always read the comment live". */ + boolean isEnabled() { + return entry.stats().isEffectiveEnabled(); + } + + /** + * Returns the cached comment for {@code identifier} if present and unexpired, else runs {@code loader} (the + * live remote {@code loadTable} + property read), caches and returns it. Disabled cache -> {@code loader} + * every call. The loader runs OUTSIDE Caffeine's compute lock (single-flight per key) and its exception + * propagates unwrapped and is NOT cached (the caller degrades a thrown comment load to {@code ""}). + */ + String getOrLoad(TableIdentifier identifier, Supplier loader) { + return entry.get(identifier, ignored -> loader.get()); + } + + /** Drops the cached comment for one table so the next read goes live (REFRESH TABLE). */ + void invalidate(TableIdentifier identifier) { + entry.invalidateKey(identifier); + } + + /** Drops every cached comment for one database (REFRESH DATABASE / DROP DATABASE); match = namespace equality. */ + void invalidateDb(String dbName) { + Namespace ns = Namespace.of(dbName); + entry.invalidateIf(id -> id.namespace().equals(ns)); + } + + /** Drops all cached comments (REFRESH CATALOG). */ + void invalidateAll() { + entry.invalidateAll(); + } + + /** Test-only: current number of cached entries (accurate map membership, not Caffeine's estimate). */ + int size() { + int[] count = {0}; + entry.forEach((key, value) -> count[0]++); + return count[0]; + } + + /** Test-only: how many times the live loader (the remote comment load) actually ran — the metric gate. */ + long loadCountForTest() { + return entry.stats().getLoadSuccessCount(); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java index 88a0a788a2a98b..172bfba369024d 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java @@ -172,6 +172,11 @@ public class IcebergConnector implements Connector { // (table, snapshotId)). Like partitionCache, the value is pure metadata (a format-name string) with no // FileIO/credential, so no gate; only the TTL knob disables it. private final IcebergFormatCache formatCache; + // PERF-05: cross-query table-comment cache (value = the 'comment' property string). Built ONLY for a REST + // vended-credentials catalog that is NOT session=user -- plain catalogs already reuse tableCache (PERF-01) for + // the comment path, and session=user must stay live because the loadTable itself carries per-user + // authorization a shared cache would bypass. null for every other flavor. + private final IcebergCommentCache commentCache; private final IcebergManifestCache manifestCache = new IcebergManifestCache(); // commit-bridge supply (S4 part 2): per-catalog stash carrying a row-level DML's non-equality delete supply // across the scan->write seam — the scan provider fills it (keyed by queryId), the write provider drains it @@ -219,6 +224,16 @@ public IcebergConnector(Map properties, ConnectorContext context // PERF-03: inferred-file-format cache. Pure metadata (no credentials) -> no gate; same TTL/capacity. this.formatCache = new IcebergFormatCache( resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY); + // PERF-05: table-comment cache, built ONLY for a REST vended-credentials catalog that is NOT session=user. + // Plain catalogs (tableCache on) already serve the comment path from tableCache; session=user is excluded + // because a shared comment cache would bypass the per-user loadTable authorization (a metadata disclosure). + // Comment is pure metadata (no credential) so no gate on the VALUE -- the flavor gate is an authorization + // decision, not a credential-leak one. Same TTL/capacity (ttl<=0 still disables internally). + this.commentCache = (IcebergScanPlanProvider.restVendedCredentialsEnabled(this.properties) + && !isUserSessionEnabled()) + ? new IcebergCommentCache( + resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY) + : null; } /** @@ -242,8 +257,8 @@ static long resolveTableCacheTtlSecond(Map properties) { @Override public ConnectorMetadata getMetadata(ConnectorSession session) { - return new IcebergConnectorMetadata( - newCatalogBackedOps(session), properties, context, latestSnapshotCache, tableCache, partitionCache); + return new IcebergConnectorMetadata(newCatalogBackedOps(session), properties, context, + latestSnapshotCache, tableCache, partitionCache, commentCache); } /** @@ -559,6 +574,9 @@ public void invalidateTable(String dbName, String tableName) { } partitionCache.invalidate(TableIdentifier.of(dbName, tableName)); formatCache.invalidate(TableIdentifier.of(dbName, tableName)); + if (commentCache != null) { + commentCache.invalidate(TableIdentifier.of(dbName, tableName)); + } } /** @@ -580,6 +598,9 @@ public void invalidateDb(String dbName) { } partitionCache.invalidateDb(dbName); formatCache.invalidateDb(dbName); + if (commentCache != null) { + commentCache.invalidateDb(dbName); + } } /** @@ -597,6 +618,9 @@ public void invalidateAll() { } partitionCache.invalidateAll(); formatCache.invalidateAll(); + if (commentCache != null) { + commentCache.invalidateAll(); + } manifestCache.invalidateAll(); } @@ -641,6 +665,11 @@ IcebergFormatCache formatCacheForTest() { return formatCache; } + /** Test-only: the table-comment cache (PERF-05), or {@code null} unless vended-credentials and non-session. */ + IcebergCommentCache commentCacheForTest() { + return commentCache; + } + @Override public ConnectorScanPlanProvider getScanPlanProvider() { // Mirrors PaimonConnector.getScanPlanProvider: build a fresh provider per call over the lazily-built diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index f24ad5541d9b08..3aadcd2ff4e7b1 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -151,6 +151,10 @@ public class IcebergConnectorMetadata implements ConnectorMetadata { // PERF-02: cross-query partition-view cache (null = no cross-query layer; the convenience ctors used by // direct-construction tests pass null). Consumed by getMvccPartitionView / listPartitions / listPartitionNames. private final IcebergPartitionCache partitionCache; + // PERF-05: cross-query table-comment cache (null = no cross-query layer). Non-null only when the owning + // connector is a REST vended-credentials, non-session catalog (see IcebergConnector); the convenience ctors + // used by direct-construction tests pass null. Consumed only by getTableComment. + private final IcebergCommentCache commentCache; public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties, ConnectorContext context) { @@ -173,12 +177,21 @@ public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties, ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache, IcebergTableCache tableCache, IcebergPartitionCache partitionCache) { + this(catalogOps, properties, context, latestSnapshotCache, tableCache, partitionCache, null); + } + + /** Full ctor used by {@link IcebergConnector#getMetadata}, adding the PERF-05 table-comment cache. */ + public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map properties, + ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache, + IcebergTableCache tableCache, IcebergPartitionCache partitionCache, + IcebergCommentCache commentCache) { this.catalogOps = catalogOps; this.properties = properties; this.context = context; this.latestSnapshotCache = latestSnapshotCache; this.tableCache = tableCache; this.partitionCache = partitionCache; + this.commentCache = commentCache; } // ========== ConnectorSchemaOps ========== @@ -333,6 +346,19 @@ public String getTableComment(ConnectorSession session, String dbName, String ta // Comment column would all be blank even though the raw comment key still appears in the SHOW CREATE // PROPERTIES(...) block (F9/F12). Views render their comment through getViewDefinition / the view // SHOW CREATE arm, so a view handle here (loadTable throws) falls back to "" via the caller's catch. + // PERF-05: on a vended-credentials (non-session) catalog, memoize the comment per table across queries so + // the per-table loadTable that information_schema.tables / SHOW TABLE STATUS pays for EVERY table collapses + // on repeats. commentCache is null for every other flavor (plain catalogs reuse tableCache via loadTable; + // session=user must stay live to preserve per-user authorization) -> resolve directly. A thrown load (view + // handle) is not cached and propagates to the caller's catch (still ""), so behavior is unchanged. + if (commentCache != null) { + return commentCache.getOrLoad(TableIdentifier.of(dbName, tableName), + () -> loadTableComment(dbName, tableName)); + } + return loadTableComment(dbName, tableName); + } + + private String loadTableComment(String dbName, String tableName) { Table table = loadTable(new IcebergTableHandle(dbName, tableName)); return table.properties().getOrDefault(TABLE_COMMENT_PROP, ""); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCommentCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCommentCacheTest.java new file mode 100644 index 00000000000000..e8a6f21119e95e --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCommentCacheTest.java @@ -0,0 +1,177 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.iceberg.catalog.TableIdentifier; +import org.apache.iceberg.exceptions.NoSuchTableException; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Unit tests for {@link IcebergCommentCache} (PERF-05). Mirrors {@link IcebergTableCacheTest} but stores the + * {@code comment} string keyed by {@link TableIdentifier}. Covers within-TTL stability, the {@code ttl <= 0} + * disable (load-bearing: a vended no-cache catalog builds this object but must not cache), invalidation, and the + * not-cached-on-failure guarantee that keeps the view-handle {@code NoSuchTableException} degradation to "". + */ +public class IcebergCommentCacheTest { + + private static TableIdentifier id(String db, String tbl) { + return TableIdentifier.of(db, tbl); + } + + @Test + public void cachesWithinTtlAndServesTheSameComment() { + AtomicInteger loads = new AtomicInteger(); + IcebergCommentCache c = new IcebergCommentCache(100, 1000); + + String first = c.getOrLoad(id("db", "t"), () -> { + loads.incrementAndGet(); + return "sales fact"; + }); + // Second read of the SAME table within TTL returns the cached comment, NOT a fresh remote loadTable -> this + // is what collapses the per-table information_schema loads on repeat queries. MUTATION: loading live every + // call -> returns "other" / loads==2 -> red. + String second = c.getOrLoad(id("db", "t"), () -> { + loads.incrementAndGet(); + return "other"; + }); + Assertions.assertEquals("sales fact", first); + Assertions.assertEquals("sales fact", second, "within TTL the cached comment must be served"); + Assertions.assertEquals(1, loads.get(), "the live loadTable must run exactly once within TTL"); + Assertions.assertEquals(1, c.loadCountForTest(), "the metric-gate load count must be 1"); + Assertions.assertTrue(c.isEnabled()); + } + + @Test + public void differentTableIsADifferentKey() { + AtomicInteger loads = new AtomicInteger(); + IcebergCommentCache c = new IcebergCommentCache(100, 1000); + c.getOrLoad(id("db", "t1"), () -> { + loads.incrementAndGet(); + return "c1"; + }); + String t2 = c.getOrLoad(id("db", "t2"), () -> { + loads.incrementAndGet(); + return "c2"; + }); + Assertions.assertEquals("c2", t2); + Assertions.assertEquals(2, loads.get()); + Assertions.assertEquals(2, c.size()); + } + + @Test + public void ttlZeroDisablesCachingAlwaysLive() { + AtomicInteger loads = new AtomicInteger(); + IcebergCommentCache c = new IcebergCommentCache(0, 1000); + c.getOrLoad(id("db", "t"), () -> { + loads.incrementAndGet(); + return "a"; + }); + String second = c.getOrLoad(id("db", "t"), () -> { + loads.incrementAndGet(); + return "b"; + }); + // ttl-second=0 (a vended no-cache catalog) loads live every time -> operator "no meta cache" intent honored + // even though the connector built this object. MUTATION: caching despite ttl<=0 -> second=="a" / loads==1. + Assertions.assertEquals("b", second, "ttl-second=0 must always load live"); + Assertions.assertEquals(2, loads.get()); + Assertions.assertFalse(c.isEnabled()); + Assertions.assertEquals(0, c.size(), "ttl-second=0 must not store anything"); + } + + @Test + public void negativeTtlDisablesCachingAlwaysLive() { + AtomicInteger loads = new AtomicInteger(); + IcebergCommentCache c = new IcebergCommentCache(-1, 1000); + c.getOrLoad(id("db", "t"), () -> { + loads.incrementAndGet(); + return "a"; + }); + String second = c.getOrLoad(id("db", "t"), () -> { + loads.incrementAndGet(); + return "b"; + }); + Assertions.assertEquals("b", second, "ttl-second=-1 must always load live"); + Assertions.assertEquals(2, loads.get()); + Assertions.assertFalse(c.isEnabled()); + } + + @Test + public void invalidateForcesReload() { + AtomicInteger loads = new AtomicInteger(); + IcebergCommentCache c = new IcebergCommentCache(100, 1000); + c.getOrLoad(id("db", "t"), () -> { + loads.incrementAndGet(); + return "old"; + }); + // REFRESH TABLE db.t drops the entry so an external ALTER ... SET TBLPROPERTIES('comment'=...) is picked up. + // MUTATION: invalidate not clearing the key -> "old" survives -> red. + c.invalidate(id("db", "t")); + String after = c.getOrLoad(id("db", "t"), () -> { + loads.incrementAndGet(); + return "new"; + }); + Assertions.assertEquals("new", after); + Assertions.assertEquals(2, loads.get()); + } + + @Test + public void invalidateDbClearsOnlyThatDbsTables() { + IcebergCommentCache c = new IcebergCommentCache(100, 1000); + c.getOrLoad(id("db1", "t1"), () -> "a"); + c.getOrLoad(id("db1", "t2"), () -> "b"); + c.getOrLoad(id("db2", "t1"), () -> "c"); + Assertions.assertEquals(3, c.size()); + c.invalidateDb("db1"); + Assertions.assertEquals(1, c.size(), "only db2's entry must survive REFRESH DATABASE db1"); + } + + @Test + public void invalidateAllClearsEverything() { + IcebergCommentCache c = new IcebergCommentCache(100, 1000); + c.getOrLoad(id("db", "t1"), () -> "a"); + c.getOrLoad(id("db", "t2"), () -> "b"); + Assertions.assertEquals(2, c.size()); + c.invalidateAll(); + Assertions.assertEquals(0, c.size()); + } + + @Test + public void loaderFailureIsNotCachedSoNextQueryRetries() { + // A view handle (loadTable throws NoSuchTableException) must NOT be cached, so getTableComment keeps + // degrading to "" via the caller's catch on every call (and a real table appearing later loads fresh). The + // MetaCacheEntry manual-miss-load path re-throws the loader's exception verbatim and does not store it. + // MUTATION: caching on failure -> the second call would not re-run the loader / size==1 -> red. + IcebergCommentCache c = new IcebergCommentCache(100, 1000); + AtomicInteger loads = new AtomicInteger(); + Assertions.assertThrows(NoSuchTableException.class, () -> c.getOrLoad(id("db", "v"), () -> { + loads.incrementAndGet(); + throw new NoSuchTableException("not a table: db.v"); + })); + Assertions.assertEquals(0, c.size(), "a thrown comment load must not be cached"); + String ok = c.getOrLoad(id("db", "v"), () -> { + loads.incrementAndGet(); + return "now a table"; + }); + Assertions.assertEquals("now a table", ok); + Assertions.assertEquals(2, loads.get(), "the loader must re-run after a failure (no sticky failure)"); + Assertions.assertEquals(1, c.size()); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java index 61465b43abd3c3..00db5d93acbd47 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java @@ -344,4 +344,63 @@ public void refreshHooksInvalidateFormatCache() { connector.invalidateAll(); Assertions.assertEquals(0, cache.size(), "REFRESH CATALOG drops everything"); } + + private static Map restProps(boolean vended, boolean sessionUser) { + Map m = new HashMap<>(); + m.put(IcebergConnectorProperties.ICEBERG_CATALOG_TYPE, IcebergConnectorProperties.TYPE_REST); + if (vended) { + m.put(IcebergConnectorProperties.REST_VENDED_CREDENTIALS_ENABLED, "true"); + } + if (sessionUser) { + m.put(IcebergConnectorProperties.REST_SESSION, IcebergConnectorProperties.SESSION_USER); + } + return m; + } + + private static IcebergCommentCache commentCacheOf(Map props) { + return new IcebergConnector(props, new RecordingConnectorContext()).commentCacheForTest(); + } + + @Test + public void commentCacheBuiltOnlyForVendedNonSessionCatalog() { + // PERF-05: the comment cache fills the gap PERF-01's tableCache leaves for vended-credentials catalogs, but + // ONLY when NOT session=user -- a session=user comment cache would serve one user's comment to another + // whose per-user loadTable authorization was never checked (a metadata disclosure). Plain catalogs already + // reuse tableCache for the comment path, so no comment cache there either. + // Plain catalog -> null (tableCache covers the comment path; no redundant cache). + Assertions.assertNull(commentCacheOf(Collections.emptyMap()), + "a plain catalog must NOT build the comment cache (tableCache already serves it)"); + // Vended, non-session -> built (the one flavor it is safe + useful for). + Assertions.assertNotNull(commentCacheOf(restProps(true, false)), + "a vended-credentials (non-session) catalog must build the comment cache"); + // session=user -> null (per-user authorization must not be bypassed by a shared cache). + Assertions.assertNull(commentCacheOf(restProps(false, true)), + "a session=user catalog must NOT build the comment cache (per-user authz)"); + // vended AND session=user -> null (session=user wins; !isUserSessionEnabled() gates it off). + Assertions.assertNull(commentCacheOf(restProps(true, true)), + "vended + session=user must NOT build the comment cache (session=user takes precedence)"); + } + + @Test + public void refreshHooksInvalidateCommentCache() { + // The REFRESH hooks must clear the comment cache too (else an external ALTER comment stays invisible beyond + // the pin): REFRESH TABLE drops that table, REFRESH DATABASE that db, REFRESH CATALOG everything. MUTATION: + // an invalidate* hook not touching commentCache -> a stale comment survives -> a size assert below red. + IcebergConnector connector = new IcebergConnector(restProps(true, false), new RecordingConnectorContext()); + IcebergCommentCache cache = connector.commentCacheForTest(); + Assertions.assertNotNull(cache); + cache.getOrLoad(TableIdentifier.of("db1", "t1"), () -> "c1"); + cache.getOrLoad(TableIdentifier.of("db1", "t2"), () -> "c2"); + cache.getOrLoad(TableIdentifier.of("db2", "t1"), () -> "c3"); + Assertions.assertEquals(3, cache.size()); + + connector.invalidateTable("db1", "t1"); + Assertions.assertEquals(2, cache.size(), "REFRESH TABLE drops db1.t1"); + + connector.invalidateDb("db1"); + Assertions.assertEquals(1, cache.size(), "REFRESH DATABASE drops db1's remaining table"); + + connector.invalidateAll(); + Assertions.assertEquals(0, cache.size(), "REFRESH CATALOG drops everything"); + } } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java index 60e4b47c718951..876e84d9a55a24 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java @@ -156,6 +156,34 @@ private static IcebergConnectorMetadata metadataWithTableProps(Map each call reloads -> loadCountForTest > 1 / >1 remote "loadTable:" -> red. + Map props = new HashMap<>(); + props.put("comment", "sales fact"); + RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); + ops.table = new FakeIcebergTable( + "t1", idNameSchema(), PartitionSpec.unpartitioned(), "s3://bucket/db1/t1", props); + IcebergCommentCache cache = new IcebergCommentCache(100, 1000); + IcebergConnectorMetadata metadata = new IcebergConnectorMetadata(ops, Collections.emptyMap(), + new RecordingConnectorContext(), new IcebergLatestSnapshotCache(0L, 1), null, null, cache); + + Assertions.assertEquals("sales fact", metadata.getTableComment(null, "db1", "t1")); + Assertions.assertEquals("sales fact", metadata.getTableComment(null, "db1", "t1")); + Assertions.assertEquals("sales fact", metadata.getTableComment(null, "db1", "t1")); + + Assertions.assertEquals(1, cache.loadCountForTest(), + "repeated getTableComment at one table must load the comment exactly once"); + long remoteLoads = ops.log.stream().filter(s -> s.startsWith("loadTable:db1.t1")).count(); + Assertions.assertEquals(1, remoteLoads, "exactly one remote loadTable across the repeats"); + // Parity: the cached value equals a direct (uncached) read of the comment property. + Assertions.assertEquals("sales fact", + metadataWithTableProps(props).getTableComment(null, "db1", "t1")); + } + private static void assertCopyOnWriteRejected( IcebergConnectorMetadata md, WriteOperation op, String operationLabel, String property) { DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, From 57e78f4fd7ddc54d8e5300a5b58d891014d31931 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 10:42:26 +0800 Subject: [PATCH 13/47] [doc](catalog) fe-connector-iceberg PERF-05: design + summary + tracker updates; next = PERF-06 Records the vended-catalog table-comment cache (PERF-05, commit aea3ebdd40e): design doc (with the re-scoping vs PERF-01 and the red-team security fix that narrows the gate to vended-non-session), summary, tasklist check-off, progress log, and a fresh HANDOFF pointing at PERF-06 (the per-file vended StorageProperties rebuild, C3) with the fe-core DefaultConnectorContext routing decision flagged. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 52 ++++++------ .../FIX-PERF-05-comment-cache-design.md | 79 +++++++++++++++++++ .../FIX-PERF-05-comment-cache-summary.md | 39 +++++++++ plan-doc/perf-hotpath-iceberg/progress.md | 14 ++++ plan-doc/perf-hotpath-iceberg/tasklist.md | 10 ++- 5 files changed, 164 insertions(+), 30 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-05-comment-cache-design.md create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-05-comment-cache-summary.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index ea8618aa2efe47..7eb04a68b5364c 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -6,49 +6,49 @@ --- -# ✅ 已完成（PERF-01 ~ 04） +# ✅ 已完成（PERF-01 ~ 05） -- **PERF-01**（`484f0e0c125`）：胖 handle（查询内）+ 跨查询 `IcebergTableCache`（**凭证 gate**）。一次规划远端 loadTable 3~7→1。 -- **PERF-02**（`518d0599cbf`）：跨查询 `IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、**无 gate**）。分区表分析期 PARTITIONS 扫描每查询→每快照一遍。 -- **PERF-03**（`0b96f2e6c78`）：跨查询 `IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、值格式名、**无 gate**）。无 write-format 表整表格式推断每查询→每快照一遍。 -- **PERF-04**（`2e5f393779c`）：抽惰性 `cacheBackedFileScanTasks`（delete 索引 eager + data manifest 惰性扁平映射）三处复用，把 manifest cache 接回大表流式 + COUNT(*)（**缓存与不 OOM 兼得**，否决审计"退回物化"因重引 legacy OOM）。小结 `designs/FIX-PERF-04-*-summary.md`。 -- 四轮全 iceberg 模块单测绿（932→943→957→962 / 0 fail / 1 skip），checkstyle 绿，0 回归。 -- **可复用产物**：三套 `(TableIdentifier[,snapshotId])` + `MetaCacheEntry` 缓存基建（table/partition/format）；**凭证 gate 判据**（值含 FileIO/凭证→gate，纯元数据→无 gate，已三次印证）；「缓存只存 loader 原始输出、映射/抛点放 getOrLoad 之外、失败透传 unchecked 不入缓存」范式；`loadCountForTest`/`cache.size()` 度量守门法；「delete 索引 eager + 惰性扁平映射」范式。 +- **PERF-01**（`484f0e0c125`）：胖 handle + 跨查询 `IcebergTableCache`（**凭证 gate**）。规划期 loadTable 3~7→1。 +- **PERF-02**（`518d0599cbf`）：`IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、无 gate）。分区扫描每查询→每快照。 +- **PERF-03**（`0b96f2e6c78`）：`IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、无 gate）。格式推断每查询→每快照。 +- **PERF-04**（`2e5f393779c`）：抽惰性 `cacheBackedFileScanTasks` 三处复用，manifest cache 接回大表流式 + COUNT(*)（缓存与不 OOM 兼得）。 +- **PERF-05**（`aea3ebdd40e`）：`IcebergCommentCache`（键 `TableIdentifier`、无 gate），**仅 vended 非 session=user**（红队：session=user 缓存绕过 per-user 授权=泄漏）。information_schema comment 每表 load 跨查询坍缩（vended）。小结 `designs/FIX-PERF-05-*-summary.md`。 +- 五轮全 iceberg 模块单测绿（→973 / 0 fail / 1 skip），checkstyle 绿，0 回归。 +- **可复用产物**：四套缓存基建（table/partition/format/comment）；**gate 三判据**：①值含 FileIO/凭证→gate；②纯元数据→无 gate；③**授权发生在 load 调用里（session=user）→即便值无凭证也不可共享缓存**（PERF-05 新增，与①②正交叠加）。「缓存只存 loader 原始输出、失败透传不缓存」；「delete 索引 eager + 惰性扁平映射」（PERF-04）。 --- -# 🆕 下一个 session = **PERF-05（C9 information_schema.tables 每表 loadTable 只为取 comment）** +# 🆕 下一个 session = **PERF-06（C3 REST vended-credentials 每文件重建 StorageProperties+Configuration）** -## 病灶（触发面广、教科书级"伪装成轻访问器"） +## 病灶 -`FrontendServiceImpl.listTableStatus` 的 **fe-core** `for (TableIf table : tables)` 循环里**无条件**调 `status.setComment(table.getComment())`（不看请求是否要 comment 列，且在 `table.readLock()` 下**串行**）→ `PluginDrivenExternalTable.getComment`（fe-core）→ `IcebergConnectorMetadata.getTableComment`（连接器）→ **每表一次 `loadTable`**（HMS getTable RPC + metadata.json GET，或 REST loadTable）。一个库 N 张 iceberg 表 = 一条 `SELECT * FROM information_schema.tables` / `SHOW TABLE STATUS` 付 **N 次串行远端 load**（每次 50~300ms），几百表数十秒~分钟，BI 工具高频触发。 +REST vended-credentials 目录，扫描规划期 `buildRange → normalizeUri → context.normalizeStorageUri(rawUri, vendedToken)`（`TcclPinningConnectorContext` → **fe-core** `DefaultConnectorContext.normalizeStorageUri:392-409`）→ `buildVendedStorageMap:225-242`（`CredentialUtils.filterCloudStorageProperties` + `StorageProperties.createAll`：遍历所有 provider + **建 hadoop `Configuration` + 逐 key set**）。**每 data-file range 一次 + 每 MOR delete file per task 一次 + 每 position-delete split 一次**（同步/流式/$position_deletes 三路径同）。vendedToken **整个 scan 内不变**（`planScanInternal:586-587` sync / `:458-459` streaming 只提取一次、同一 Map 实例穿所有调用），但每文件都重建整份 map+Configuration。50k 文件 ≈ **数十秒纯 FE CPU**（MOR 加倍）。门槛：REST + `iceberg.rest.vended-credentials-enabled=true`（空 token 在 `:227-229` 短路）。 ## 第一件事（立项流程见 README §单项立项流程） -1. **复核（动码前，行号信 grep 不信文档；PERF-01~04 后行号已漂移）**：grep `FrontendServiceImpl.listTableStatus`(fe-core) 的循环 + `setComment` 是否仍无条件、是否在 readLock 下；`PluginDrivenExternalTable.getComment` 是否有缓存字段；**`IcebergConnectorMetadata.getTableComment` 当前是裸 `loadTable` 还是已走 PERF-01 的 `resolveTableForRead`（tableCache）**？——这决定修法起点。 -2. **⚠ 铁律约束（关键）**：循环在 **fe-core**（`FrontendServiceImpl` / `PluginDrivenExternalTable.getComment`），**「fe-core 源只出不进」→ 不得改 fe-core 循环、不得给 fe-core 加缓存 comment 字段**。修法必须**连接器侧**。 -3. **设计**（写 `designs/FIX-PERF-05-...-design.md`），候选方向： - - **(a) 连接器侧 comment 缓存**（键 `TableIdentifier`、值 comment `String`、**无凭证 gate**——纯元数据）：`getTableComment` 先查缓存。**镜像 PERF-03 format cache**。**注意**：这消不掉**首次** N 次 load（无批量 SPI），但让**重复** information_schema 查询坍缩（BI 高频→除以 QPS-over-TTL）；REFRESH TABLE/DB/CATALOG 失效。 - - **(b) 若 getTableComment 尚未走 tableCache**：至少改走 PERF-01 的缓存 resolve，让"已被扫描加载过的表" + "重复查询"免费（与 (a) 可叠加）。 - - **批量 load（HMS `get_table_objects_by_name`）不可行**：需新 SPI 批量方法 + 改 fe-core 循环（fe-core 加法，越界）——排除，记为已知局限（首次 N load 无法在本约束下消除）。 -4. **红队 + TDD**：度量守门 = 重复取同表 comment 时远端 `loadTable`/`getTableComment` 次数从 N→命中（仿 `loadCountForTest`）；gate = 三目录（plain/vended/session）comment 缓存均建（无 gate）+ REFRESH 失效。 -5. 守铁律：缓存放连接器侧，值纯元数据无 gate；不碰 fe-core。 +1. **复核（动码前，行号信 grep；PERF-01~05 后已漂移）**：grep `buildRange`/`convertDelete`/`buildPositionDeleteRange` 的 `normalizeUri` 调用点、`DefaultConnectorContext.normalizeStorageUri` + `buildVendedStorageMap`。**核心确认**：`buildVendedStorageMap(token)` 的重活（`StorageProperties.createAll` + hadoop Configuration）是**纯 token 派生、与 rawUri 无关**吗？（审计说是——URI 只用于路径归一化那半，cheap。） +2. **⚠ 路由决策（关键，可能需上交用户，像 PERF-04/05）**：`DefaultConnectorContext` 在 **fe-core**。两条路线： + - **(a) 连接器侧 hoist**：iceberg 扫描期把 token→map 派生**提升到 scan 级**（token 已只提取一次），per-file 只做 cheap 路径归一化。**难点**：`normalizeStorageUri(uri, token)` 是 fe-core `ConnectorContext` API，重活封装在 fe-core 里；连接器要 hoist 需 fe-core 暴露"派生一次 map"的入口，或连接器自己做路径归一化——**核 `ConnectorContext` API 表面**能否连接器侧 hoist 而不改 fe-core。 + - **(b) fe-core 框架 memo**：`DefaultConnectorContext` 内按 **token 恒等键单条目 memo** `buildVendedStorageMap`。这是 fe-core **框架层**改动——按 README 铁律 2（PERF-09 先例：**改通用框架、惠及所有连接器、非 source-specific = 允许**）**大概率合规**，但属**共享热路径**，须证 **byte + cost 对所有连接器双不变**（对齐 PERF-09/共享 MVCC 纪律），且 token 作 key 要用**恒等/内容**键谨慎（同 Map 实例→identity 键最稳）。 + - **倾向**：先核 (a) 可行性；若 (a) 须改 fe-core API 才能 hoist，则 (b)（框架 memo，惠及所有 vended 连接器）反而更干净——但**两条都动到 fe-core 边界**，故**动码前把选择+理由用中文上交用户定**（不引用任务代号）。 +3. **红队 + TDD**：度量守门 = 一次 vended scan 内 `buildVendedStorageMap`/`StorageProperties.createAll` 远端/CPU 次数从 O(N_files+N_deletes)→1（可加计数器或 spy）。 +4. 守铁律：优先连接器侧；若走 fe-core 框架 memo，须 connector-agnostic（禁按源名分支）+ 双不变证明。 ## ⚠️ 关键认知 -- **缓存都挂 `IcebergConnector`**（长生命周期）；comment 缓存键用 `TableIdentifier`（comment 是表级属性，与快照无关，故**不需要 snapshotId**——与 format/partition 缓存不同，那两个随快照变，comment 随 DDL 变靠 REFRESH 失效即可）。 -- **凭证 gate 判据**：comment 是纯元数据字符串（无 FileIO/凭证）→ **无 gate**（同 partition/format 缓存）。 -- **首次 N load 是本约束下的硬下限**：诚实写进设计 Risk（不假装消除），只优化重复。 +- **token loop-invariant 是本修的根**：token 每 scan 提取一次、同一 Map 实例穿所有 `normalizeUri`；重活是 token 派生、与 per-file URI 无关 → 可 memo/hoist 一次。 +- **三路径都要覆盖**：data-file range（buildRange）+ MOR delete（convertDelete）+ position-delete split（buildPositionDeleteRange），同步与流式两条。别只修一路。 +- **gate 判据延伸**（PERF-05 新增）：本任务缓存的是 storage map（**含凭证/token 派生**）→ 若做跨 scan 缓存会撞凭证 gate；但本修是 **scan 内**复用（token 恒定），非跨查询，故**不涉凭证泄漏**——scan 内 hoist/memo 安全。 --- # 🧰 构建/验证坑（**实证，务必照做**） -1. **可靠跑单测 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`**（绝对 `-f /fe/pom.xml`）。原因：本 worktree `${revision}` + 未 flatten 已装 pom → `-pl iceberg` 解析不到 `fe-connector:pom:${revision}`，必须 `-am`；`-am test` 只到 test 相不产 hms-hive-shade shade jar（缺 `HiveConf`），故用 **`install`**。`-Dtest=`（=`ls src/test/.../*Test.java` 拼逗号，本轮 55 类，存 `scratchpad/iceberg-tests.txt`）让上游 0 匹配测试快跳。**checkstyle 单独** `mvn checkstyle:check -pl fe-connector/fe-connector-iceberg`。 +1. **可靠跑单测 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`**（绝对 `-f /fe/pom.xml`）。原因：`${revision}` + 未 flatten 已装 pom → `-pl iceberg` 解析不到 `fe-connector:pom:${revision}`，必须 `-am`；`-am test` 只到 test 相不产 hms-hive-shade shade jar（缺 `HiveConf`），故 **`install`**。`-Dtest=<类列表>`（`ls src/test/.../*Test.java` 拼逗号，本轮 56 类，存 `scratchpad/iceberg-tests.txt`）让上游快跳。 2. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 静默跳过）。 -3. **`mvn -q` 成功时不打 `BUILD SUCCESS`/`Tests run`（INFO 被抑制）**——本轮踩坑。验证成功用 **surefire XML 聚合**（`target/surefire-reports/TEST-*.xml` 的 tests/failures/errors 属性求和），别只看 exit code / grep BUILD 行。或去掉 `-q`。 -4. **别 `nohup ... &` 套在 Bash `run_in_background` 里**——外层立即随 echo 退出、maven 变孤儿，"completed exit 0" 是假信号。直接 `mvn ... > log 2>&1`（run_in_background）。 -5. **checkstyle**：LineLength(120) 对 test 源 suppress；主源 ≤120；import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行。**新用类记得 import**（本轮 `java.util.Iterator` 漏 import 编译挂）。 +3. **别用 `mvn -q` 跑测试**（本轮踩坑）：成功时抑制 `BUILD SUCCESS`/`Tests run` INFO 行 → 无法从日志确认。**去掉 `-q`**，或用 surefire XML 聚合（`target/surefire-reports/TEST-*.xml` 求和 tests/failures/errors）验证。 +4. **别 `nohup ... &` 套 `run_in_background`**——外层立即随 echo 退出、maven 变孤儿，"exit 0" 假信号。直接 `mvn ... > log 2>&1`（run_in_background）。 +5. **checkstyle 主源 ≤120（test 源 suppress）**：本轮 2 处 javadoc 超行踩坑。import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行；**新用类记得 import**（PERF-04 漏 `java.util.Iterator`）。checkstyle 单独 `mvn checkstyle:check -pl fe-connector/fe-connector-iceberg`。 6. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。 7. **并发探测**：本目录 `find fe -newermt '-120 sec'` + `pgrep -a -f maven`。`/mnt/disk1/yy/git/doris` 是另一 worktree，不冲突。 8. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 @@ -57,4 +57,4 @@ # 🗂 开放问题 -- **PERF-05 首次 N load 硬下限**：本约束（fe-core 不可改）下，comment 缓存只能优化重复查询，首次 `SELECT * FROM information_schema.tables` 仍付 N 次串行 load。批量 load 需改 fe-core → 越界。设计须诚实记为已知局限（对齐 PERF-02 "范围决定" 的诚实口径）。 +- **PERF-06 路由（连接器 hoist vs fe-core 框架 memo）**：`DefaultConnectorContext` 在 fe-core。须先核 `ConnectorContext` API 能否连接器侧 hoist token→map 派生；若须改 fe-core 才能 hoist，则 fe-core 框架层 token-memo（惠及所有 vended 连接器、非 source-specific，PERF-09 先例大概率合规，但须双不变证明）可能更干净。**动码前把方案上交用户定**。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-05-comment-cache-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-05-comment-cache-design.md new file mode 100644 index 00000000000000..f6f9e791b42ed5 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-05-comment-cache-design.md @@ -0,0 +1,79 @@ +# FIX-PERF-05 — Design（凭证 gated 目录的表 comment 跨查询缓存） + +> 覆盖审计发现 **C9**（information_schema.tables / SHOW TABLE STATUS 每表 loadTable 只为取 comment）。 +> 用户 2026-07-18 拍板方向 = **补注释缓存（仅特殊目录）**。基线 HEAD `25fc2bead7a`。行号信 grep。 + +## Problem（含复核后的范围缩小） + +fe-core `FrontendServiceImpl.listTableStatus` 的 `for (TableIf table : tables)` 循环**无条件**（不看请求是否要 comment 列、且在 `table.readLock()` 下**串行**）调 `table.getComment()` → `PluginDrivenExternalTable.getComment`（无缓存字段）→ `IcebergConnectorMetadata.getTableComment` → **每表一次 `loadTable`**（HMS getTable RPC + metadata.json GET / REST loadTable），只为读一句 `comment` 属性。N 表 = N 次串行远端 load。 + +**⚠ 复核（审计写于 PERF-01 之前，范围已缩小）**：`getTableComment` 现在走 `loadTable → resolveTableForRead → PERF-01 的 IcebergTableCache`（键 `TableIdentifier`，跨查询）。故： +- **普通目录**：首查 N load（顺带把 N 张 raw Table 缓进 tableCache），**重复查询命中 tableCache → 0 load**。BI 高频反复查这个最痛点，普通目录**已被 PERF-01 兜住**。 +- **凭证 gated 目录**（`iceberg.rest.session=user` / REST vended-credentials）：tableCache **被 gate 掉为 null**（缓存的 raw Table 带 per-user/vended 凭证，跨用户共享会泄漏）→ `getTableComment` 每次裸 `catalogOps.loadTable` → **无跨查询复用 → 每查询 N load**。这是 PERF-01 未覆盖的残余。 +- **任何目录首次查询**：仍 N load（消除需批量 SPI + 改 fe-core 循环 → 违反「fe-core 只出不进」→ 越界）。 + +## Root Cause & 修法方向 + +comment 是纯元数据字符串。但**能否跨用户共享取决于目录类型**（红队 HIGH 修正，见下）： +- **REST vended-credentials 目录**：单一静态目录身份，所有 Doris 用户 load 的是**同一张表**，comment 跨用户共享**安全** → 可缓存。 +- **`iceberg.rest.session=user` 目录**：授权发生在 `loadTable` 调用本身（用带查询用户 delegated token 的 per-user 目录）。**缓存命中会绕过这次 per-user 授权**：用户 A（有权）load 出 comment 入缓存，用户 B 虽有合法 token（能建 metadata）且过 Doris SHOW 权限、但其 REST 侧对该表**无权**——今天 B 的 live loadTable 抛→`""`（不泄漏），**缓存后 B 命中 A 的条目、拿到 comment 而 B 的 token 从未被校验** = 元数据泄漏，且是**相对现状（session=user 无 tableCache、次次 live per-user）的回退**。故 **session=user 不可缓存**。 + +修法：**给 comment 单独做一个轻量、无凭证 gate 的缓存，仅在 vended-credentials 且非 session=user 时启用**。恰好补 PERF-01 对 vended 目录的空缺；普通目录零冗余（tableCache 已管重复）；session=user 保持次次 live（安全优先，正确付 N load）。 + +## Design（连接器侧，零 fe-core 改动） + +### 1. 新建 `IcebergCommentCache`（挂 `IcebergConnector`） +- **镜像 `IcebergTableCache`**（键 `TableIdentifier` 直接、非复合——comment 是表级、与快照无关；`invalidateKey`/`invalidateDb`(namespace 相等)/`invalidateAll`）；**值 = comment `String`**；`isEnabled`/`getOrLoad`/`size`/`loadCountForTest`（度量守门）。 +- **无凭证 gate**：值纯元数据字符串。 +- TTL = `meta.cache.iceberg.table.ttl-second`（同其余缓存，`<=0` 关；comment 随 DDL 变靠 REFRESH 失效，无需 snapshot 键）。 + +### 2. `IcebergConnector` +- field `commentCache`；构造（**在 `this.tableCache = …` 之后**） + **`this.commentCache = (IcebergScanPlanProvider.restVendedCredentialsEnabled(this.properties) && !isUserSessionEnabled()) ? new IcebergCommentCache(ttl, cap) : null;`** + ——**仅 vended-credentials 且非 session=user 时建**（红队 HIGH：session=user 缓存会绕过 per-user 授权泄漏 comment；session=user 与 vended 同开时 `!isUserSessionEnabled()` 为 false → 不建，session=user 语义优先）。普通目录 / session=user → commentCache=null，`getTableComment` 直穿 loadTable（普通目录用 tableCache；session=user 次次 live per-user）。 +- `getMetadata` 传 `commentCache`；`invalidateTable/Db/All` 各加 `if (commentCache != null) commentCache.invalidate*`；`commentCacheForTest()`。 +- **ttl≤0 语义 load-bearing**（红队 LOW）：vended + ttl≤0 目录 commentCache 虽建但内部 disabled（镜像 `IcebergTableCache` 的 ttl≤0→disabled 映射）→ 不缓存，尊重「无 meta 缓存」意图。镜像必须保留该映射 + 保 ttl≤0 单测。 + +### 3. `IcebergConnectorMetadata` +- ctor 加 `commentCache` 参 + 字段（便利 ctor 传 null）。 +- `getTableComment`： + ``` + TableIdentifier id = TableIdentifier.of(dbName, tableName); + return commentCache != null + ? commentCache.getOrLoad(id, () -> loadTableComment(dbName, tableName)) + : loadTableComment(dbName, tableName); + ``` + 抽 `loadTableComment(db, tbl)` = 现逻辑 `loadTable(new IcebergTableHandle(db, tbl)).properties().getOrDefault(TABLE_COMMENT_PROP, "")`。 + +## Parity / 正确性 + +- **同表同 comment**：comment 是表级属性；缓存值 `String`，命中即返回，与裸 loadTable 读同一属性一致。 +- **跨用户共享安全**（**仅 vended**）：vended 目录单一静态身份，所有用户 load 同一张表 → comment 跨用户共享安全。**session=user 已排除**（其授权在 per-user loadTable 里，缓存会绕过 → 泄漏，见 Root Cause）。 +- **staleness 新窗口**（红队 LOW，诚实记账）：vended 目录本无 tableCache（次次 live），本缓存引入至多 TTL（默认 24h）的 comment 陈旧窗口（靠 REFRESH 失效），是标准缓存权衡，与普通目录对整表早已接受的一致。 +- **失败不缓存**：loader（loadTable）抛异常（如 view handle → NoSuchTable / 无 delegated token 被拒）→ `MetaCacheEntry` 不 put → 透传给 `PluginDrivenExternalTable.getComment` 的 catch → `""`（现行为不变），下次重试。 + - **View 局限**（诚实记账）：view 的 `getTableComment` loadTable 抛 → 不缓存 → 每次仍抛一次 loadTable（与今天同）；view comment 本走 `getViewDefinition` 渲染，本任务不覆盖。 +- **staleness**：ALTER 改 comment 靠 REFRESH TABLE/DB/CATALOG 失效（与 tableCache 同口径）；TTL 到期自然过期。 + +## Implementation Plan + +1. 新建 `IcebergCommentCache.java`（≈110 行，镜像 `IcebergTableCache` + `loadCountForTest`）。 +2. `IcebergConnector`：字段 + 构造（inverse gate：tableCache==null 才建）+ getMetadata 传参 + 三 invalidate（null-guard）+ `commentCacheForTest`。 +3. `IcebergConnectorMetadata`：ctor 参 + 字段 + `getTableComment` 路由 + 抽 `loadTableComment`。 + +## Test Plan + +- **单测 `IcebergCommentCacheTest`**（镜像 `IcebergTableCacheTest`/`IcebergFormatCacheTest`）：TTL 内命中同键 loadCount=1、`ttl<=0`/负值关、`invalidate`/`invalidateDb`/`invalidateAll`、loader 异常不入缓存。 +- **度量守门（集成）** `IcebergConnectorMetadataTest`（真 InMemoryCatalog / fake ops 返回带 comment 属性的表 + 直接构造带 commentCache 的 metadata）：重复 `getTableComment` 同表 → 远端 loadTable 恰 1 次（`loadCountForTest()==1`），comment 值 == 原始属性。**MUTATION**：不接缓存 → 每次 loadTable → loadCount>1 → 红。 +- **连接器 gate（vended-only）+ 失效** `IcebergConnectorCacheTest`： + - **plain 目录 → `commentCacheForTest()==null`**（tableCache 开 → comment 缓存关，无冗余）； + - **vended（非 session=user）目录 → `commentCacheForTest()!=null`**； + - **session=user 目录 → `commentCacheForTest()==null`**（红队 HIGH：per-user 授权不可绕过，保持 live）； + - **vended + session=user 同开 → `commentCacheForTest()==null`**（session=user 优先）； + - vended 目录 REFRESH TABLE/DB/CATALOG 逐级清空。 +- 全 iceberg 模块 UT 绿 + checkstyle 绿。 + +## Risk + +- **低**：连接器自包含小改动（一新小缓存 + 3 处 wiring），镜像已落地 3 套同形缓存。 +- inverse gate（tableCache==null 才建 comment 缓存）是非常规但正确的选择——普通目录已由 tableCache 覆盖，避免双缓存冗余内存；须测试锁死"plain=null / gated=非 null"。 +- 首次 N load 硬下限、view 未缓存：诚实记为已知局限，不假装消除。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-05-comment-cache-summary.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-05-comment-cache-summary.md new file mode 100644 index 00000000000000..968754fe4bbc06 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-05-comment-cache-summary.md @@ -0,0 +1,39 @@ +# FIX-PERF-05 — Summary（凭证 gated 目录的表 comment 跨查询缓存） + +> 权威设计见 [`FIX-PERF-05-comment-cache-design.md`](./FIX-PERF-05-comment-cache-design.md)。本文件只记落地结果。 +> commit `aea3ebdd40e`（`[perf]`，iceberg 自包含）。基线 HEAD `25fc2bead7a`。用户 2026-07-18 拍板 = 补注释缓存（仅特殊目录）。 + +## Problem（含范围缩小） + +fe-core `FrontendServiceImpl.listTableStatus` 每表**无条件** `getComment` → 连接器 `getTableComment` → **每表一次 loadTable** 只为读 `comment`。N 表 = N 次串行远端 load，BI 高频触发。 +**复核（审计早于 PERF-01）**：`getTableComment` 现走 `loadTable → resolveTableForRead → IcebergTableCache`（PERF-01）→ **普通目录重复查询已命中缓存**。残余 = **凭证 gated 目录**（tableCache=null）次次重载。 + +## Fix（连接器侧，零 fe-core 改动） + +- 新增**无 gate** `IcebergCommentCache`（键 `TableIdentifier`、值 comment `String`，镜像 `IcebergTableCache`）。 +- **仅当 vended-credentials 且非 session=user 时构建**（`restVendedCredentialsEnabled && !isUserSessionEnabled()`）；普通目录（tableCache 已覆盖）与 session=user 均 null。`getTableComment` 有缓存则查、否则直穿 `loadTableComment`。 +- 失败不缓存（view loadTable 抛 → 透传 caller catch → `""`，行为不变）。REFRESH TABLE/DB/CATALOG 失效。 + +## 安全范围（为何不用 `tableCache==null` 做 gate —— 红队 HIGH） + +**session=user 故意排除**：其授权在 per-user loadTable 调用本身；共享 comment 缓存会把 A 加载的 comment 发给 token 从未被校验的 B = 元数据泄漏，且是相对现状（session=user 无缓存、次次 live per-user）的**回退**。vended 单一静态身份、所有用户 load 同一表 → 共享安全。红队原判 `tableCache==null` 过宽（含 session=user），已收窄。 + +## 红队（7 攻击） + +- **HIGH**：`tableCache==null` gate 把 session=user 卷入 → 泄漏；改 vended-only（`&& !isUserSessionEnabled()`）——已修。 +- **LOW**：镜像须保 `ttl≤0→disabled` 映射（vended 无缓存目录不缓 comment）——已保 + 单测。 +- **LOW**：vended 新增 comment 陈旧窗口（≤TTL，REFRESH 失效）——记入 parity。 +- 其余（范围缩小、异常/view 契约、失效完整、键用远程名、无冗余）均 SURVIVES。 + +## Tests + +- **单测 `IcebergCommentCacheTest`**（8）：TTL 命中 loadCount=1、不同表不同键、`ttl<=0`/负值关、invalidate/DB/All、失败不缓存可重试。 +- **度量守门（集成）** `IcebergConnectorMetadataTest`：带 commentCache 的 metadata 重复 `getTableComment` 同表 → `loadCountForTest()==1` + 远端 `loadTable` 恰 1（`ops.log` 计数）+ 与直读 parity。**MUTATION**：不接缓存 → 每次重载 → 红。 +- **连接器 gate + 失效** `IcebergConnectorCacheTest`（+2）：**plain→null / vended非session→非null / session=user→null / vended+session=user→null**；vended REFRESH TABLE/DB/CATALOG 逐级清空。 + +## Result + +- 全 iceberg 模块 **973 pass / 0 fail / 0 error / 1 skip**（`install -am`），checkstyle 0 违规，0 回归。 +- 减负：**vended 目录**重复 `information_schema.tables` 从"次次 N load"→"命中缓存近 0 load"；普通目录不变（PERF-01 已覆盖）；session=user 不变（安全优先，正确付 N load）。 +- 诚实局限：任何目录**首次** N load 硬下限（不改 fe-core 消不掉）；view comment 未缓存。 +- 可复用：第 4 套 `TableIdentifier`-keyed MetaCacheEntry 缓存；**「gate 是授权决策非凭证泄漏决策」判据**（session=user 授权在 load 调用里，缓存会绕过 → 与「值含凭证才 gate」正交，二者叠加）。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index 3bf59aead1e4e5..8f1f30252fd2de 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -71,3 +71,17 @@ - **踩坑（CI 前本地捕获）**：① `java.util.Iterator` 漏 import（编译挂）；② 重构后同步 `planFileScanTaskWithManifestCache` 顶部无条件 `session.getQueryId()`，而既有空表用例传 **null session**（旧码 snapshot==null 早返回、从不碰 session）→ NPE；改 `statsQueryId = session != null ? getQueryId() : null`。 - **结果**：全 iceberg 模块 **962 pass / 0 fail / 0 error / 1 skip**，checkstyle 0 违规，0 回归（现有测试全绿）。summary 见 `designs/FIX-PERF-04-*-summary.md`。 - **下一步**：见 HANDOFF —— PERF-05（C9 information_schema.tables 每表 loadTable 只为取 comment）。 + +--- + +## 2026-07-18 — session 3（续）：PERF-05 实现 + 全绿（commit `aea3ebdd40e`） + +- **复核缩小（关键）**：审计 C9 写于 PERF-01 之前。核当前代码：`getTableComment → loadTable → resolveTableForRead → IcebergTableCache`（PERF-01）→ **普通目录重复 information_schema 查询已命中缓存**（"BI 反复查反复慢"最痛点普通目录已解）。残余 = **凭证 gated 目录**（tableCache 因带凭证被 gate 为 null）次次重载。→ 上交用户决策（AskUserQuestion，A 补注释缓存 / B 判定基本修复）。 +- **用户选 A（补注释缓存，仅特殊目录）**。设计：无 gate `IcebergCommentCache`（键 TableIdentifier、值 comment String），仅在 tableCache 关掉的凭证 gated 目录建。 +- **设计红队（独立 agent，7 攻击）抓 1 HIGH（关键安全）**：原设计 gate=`tableCache==null` **过宽**——把 **session=user** 卷入。session=user 的授权在 per-user loadTable 调用本身；共享 comment 缓存会把用户 A 加载的 comment 发给"有合法 token+过 Doris 权限、但 REST 侧对该表无权"的用户 B（B 的 token 从未被校验）= **元数据泄漏，且相对现状回退**。→ 收窄 gate 为 **`restVendedCredentialsEnabled && !isUserSessionEnabled()`**（vended 单一静态身份共享安全；session=user 保持 live）。另 2 LOW（ttl≤0→disabled 映射 load-bearing；vended 新增陈旧窗口）并入。其余 6 攻击 SURVIVES。 +- **实现（连接器侧一个 `[perf]` commit）**：新增 `IcebergCommentCache`（镜像 `IcebergTableCache` + `loadCountForTest`）；`IcebergConnector` 字段 + vended-only 构造 + getMetadata 传参 + 三 invalidate（null-guard）+ `commentCacheForTest`；`IcebergConnectorMetadata` 7 参 ctor + `getTableComment` 路由 + 抽 `loadTableComment`。 +- **测试**：`IcebergCommentCacheTest`（8：TTL/关/失效/失败不缓存）；`IcebergConnectorMetadataTest` 度量守门（`loadCountForTest()==1` + `ops.log` 远端 load 计数）；`IcebergConnectorCacheTest`（+2：gate=plain null / vended非session 非null / session=user null / vended+session null；vended REFRESH 失效）。 +- **踩坑（CI 前本地捕获）**：2 处主源行超 120（javadoc）→ checkstyle 挂 → 缩短文案（纯 docs，无行为变）。 +- **结果**：全 iceberg 模块 **973 pass / 0 fail / 0 error / 1 skip**，checkstyle 0 违规，0 回归。summary 见 `designs/FIX-PERF-05-*-summary.md`。 +- **新判据（可复用）**：**「缓存 gate 是授权决策，不止凭证泄漏决策」**——session=user 的授权发生在 load 调用里，缓存命中会绕过它，故即便缓存值不含凭证也不能对 session=user 共享；与「值含 FileIO/凭证才 gate」正交，二者叠加判定。 +- **下一步**：见 HANDOFF —— PERF-06（C3 REST vended-credentials 每 data/delete file 重建 StorageProperties+Configuration）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index 3f90f9c03954f7..a769d116a9814a 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -19,7 +19,7 @@ | PERF-02 | P0 | C7 C22 C23 | 分区视图每查询重扫 PARTITIONS 元数据表 → `(table,snapshotId)` 缓存(连接器侧,无凭证 gate);MTMV refresh pin 判定为多余(靠 latestSnapshotCache 稳定快照坍缩,不改 fe-core) | 与 01 共享快照 pin 机制 | ✅ 完成 | `518d0599cbf` | | PERF-03 | P0 | C2 C11 | #64134 复活：`file_format_type` 兜底走整表 planFiles → 跨查询 `(table,snapshotId)` memoize(连接器侧,无 gate);~~从枚举反推~~已否决(getFileFormatType 早于 planScan + 无过滤 vs 带谓词破 parity) | 与 01/02 共享快照 pin | ✅ 完成 | `0b96f2e6c78` | | PERF-04 | P1 | C17 C18 | streaming / COUNT(*) 下推旁路 IcebergManifestCache → 抽惰性 `cacheBackedFileScanTasks`(delete 索引 eager + data manifest 惰性扁平映射)三处复用;缓存与不 OOM 兼得(否决审计"退回物化"因重引 OOM) | — | ✅ 完成 | `2e5f393779c` | -| PERF-05 | P1 | C9 | information_schema.tables 循环内每表 loadTable（只为拿 comment） → 随表缓存 / 惰性取 | — | ⏳ | | +| PERF-05 | P1 | C9 | information_schema.tables 每表 loadTable 取 comment → 复核缩小(普通目录 PERF-01 已覆盖);补**无 gate** `IcebergCommentCache` **仅 vended 非 session=user**(红队:session=user 缓存绕过 per-user 授权=泄漏) | — | ✅ 完成 | `aea3ebdd40e` | | PERF-06 | P1 | C3 | REST vended-cred 每 data/delete file 重建 StorageProperties+Configuration → scan 级 memo | — | ⏳ | | | PERF-07 | P1 | C20 | 一条 DML 3~5 次 load 同表 → 语句级 resolve 一次传递 | — | ⏳ | | | PERF-08 | P2 | C19 C21 | 维护路径逐单位重扫 / 无去重（rewrite_data_files 每 group planFiles；expire_snapshots S×M）→ union 一次注册 / 按 path 去重 | 改动小可先行 | ⏳ | | @@ -59,9 +59,11 @@ - **落地**：抽惰性 `cacheBackedFileScanTasks`（delete 索引 eager + data manifest 惰性扁平映射，产整文件任务不攒整表），三处复用（同步物化它保启发式切片；流式喂它保 OOM 安全+固定切片+现在命中缓存；COUNT 迭代取首文件惰性早停）。决策不改；失败 `catch(Exception)` 退 SDK；`statsQueryId` 可空(流式 null 消跨线程 stats 竞争)。红队 7 攻击核心 sound，采纳 catch 类型/线程/测试 3 修正。 - **收益**：开缓存大表流式从 0 命中→命中且不 OOM；COUNT 从全提交→首文件即停。全模块 962 UT 绿。 -### [ ] PERF-05 — C9：information_schema.tables 每表 loadTable -- **病灶**：`FrontendServiceImpl.listTableStatus:719 for(table)` → `:755 setComment(table.getComment())`（**无条件**，不看请求是否要 comment 列）→ `PluginDrivenExternalTable.getComment:944 → IcebergConnectorMetadata.getTableComment:305 → loadTable`。N 表 = N 次串行远程 load；几百表的库一条 `SELECT * FROM information_schema.tables` 数十秒~分钟，BI 工具高频触发。教科书级"伪装成轻访问器"。 -- **修复方向**：建表/schema-cache 时捕获 comment 随表对象缓存，或该路径按需惰性取。 +### [x] PERF-05 — C9：information_schema.tables 每表 loadTable 取 comment · ✅ `aea3ebdd40e` +- **病灶**：`FrontendServiceImpl.listTableStatus` fe-core 循环每表**无条件** `getComment` → 连接器 `getTableComment` → 每表一次 loadTable 只为读 `comment`。N 表 = N 串行远端 load，BI 高频。 +- **复核缩小（审计早于 PERF-01）**：`getTableComment` 现走 `resolveTableForRead → IcebergTableCache` → **普通目录重复查询已命中**；残余 = 凭证 gated 目录（tableCache=null）。 +- **落地**：无 gate `IcebergCommentCache`（键 TableIdentifier、值 comment String），**仅 vended 且非 session=user 时建**。**红队 HIGH**：`tableCache==null` 做 gate 会卷入 session=user，其授权在 per-user loadTable 里、缓存绕过=元数据泄漏 → 收窄为 vended-only。首次 N load / view 未缓存记为诚实局限。 +- **收益**：vended 目录重复 information_schema 从次次 N load→命中；普通/session=user 不变。全模块 973 UT 绿。 ### [ ] PERF-06 — C3：vended-credentials 每文件重建 StorageProperties - **病灶**：`buildRange:1105 normalizeUri` / `convertDelete:1157 → DefaultConnectorContext.normalizeStorageUri:392-409 → buildVendedStorageMap:225-242 → StorageProperties.createAll`（遍历所有 provider + 建 hadoop `Configuration` + 逐 key set），**每 data file 和每 delete file 各一次**，而 vended token 整个 scan 内不变。50k 文件 ≈ 数十秒纯 FE CPU。门槛：REST + `iceberg.rest.vended-credentials-enabled=true`（MOR 加倍）。 From 3412f523a24d2f808ca33410de18fa99c0f011ff Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 11:52:11 +0800 Subject: [PATCH 14/47] [perf](catalog) fe-connector-iceberg: derive the vended storage config once per scan, not per file (PERF-06) Root cause: for a REST catalog with vended credentials, iceberg scan planning re-derived the whole StorageProperties map (StorageProperties.createAll + a hadoop Configuration build) for EVERY data file and EVERY delete file, via context.normalizeStorageUri(uri, token). But the vended token is scan-invariant (extracted once, the same Map instance threads through every range builder), and the derivation is a pure function of the token, independent of the per-file URI. So a 50k-file scan burned tens of seconds of FE CPU (doubled on merge-on-read) rebuilding an identical map O(N_files + N_deletes) times. Fix (connector-side "prepare once, apply many"): add a scan-scoped SPI seam ConnectorContext.newStorageUriNormalizer(token) that returns a UnaryOperator which bakes the token-derived config in once and then normalizes each URI cheaply. DefaultConnectorContext overrides it to build the effective map lazily-memoized on the first non-empty URI, byte-identical to normalizeStorageUri(uri, token) (same empty-uri short-circuit, same vended-replaces-static precedence, same fail-loud LocationPath, same exception timing). The SPI default folds back to a per-call normalizeStorageUri, so every other connector is unaffected. The iceberg provider builds the normalizer once at each of the three scan-start sites (synchronous data, streaming, position_deletes) and threads it through buildRange / convertDelete / buildPositionDeleteRange in place of the token; TcclPinningConnectorContext delegates the new method to the raw context so iceberg gets the hoist (this path is pure fe-core, no TCCL pin needed). DefaultConnectorContext is per-catalog and shared across queries, so this scan-scoped hoist deliberately keeps no cross-query state and retains no credentials (the normalizer is a scan-local, single-threaded, GC'd at scan end). Not in scope: the write path getBackendFileType is O(1) per write, not per data file. Measurement / guard: one vended scan now derives the config once instead of O(N_files + N_deletes) times. A new connector-side guard test asserts a 3-file scan enters newStorageUriNormalizer exactly once while normalizing all three paths; new fe-core parity tests assert the normalizer's output is byte-identical to normalizeStorageUri across the vended / static / empty-uri / fail-loud cases. Tests: fe-connector-iceberg full module 974 UT green (0 fail / 1 skip); fe-core DefaultConnectorContextNormalizeUriTest 13 UT green. Checkstyle clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../iceberg/IcebergScanPlanProvider.java | 81 ++++++++++--------- .../iceberg/TcclPinningConnectorContext.java | 10 +++ .../iceberg/IcebergScanPlanProviderTest.java | 48 ++++++++--- .../iceberg/RecordingConnectorContext.java | 14 ++++ .../doris/connector/spi/ConnectorContext.java | 23 ++++++ .../connector/DefaultConnectorContext.java | 32 ++++++++ ...faultConnectorContextNormalizeUriTest.java | 54 +++++++++++++ 7 files changed, 216 insertions(+), 46 deletions(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index d5124dcd9c4995..e94cd0784a5cca 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -106,6 +106,7 @@ import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.function.Function; +import java.util.function.UnaryOperator; /** * {@link ConnectorScanPlanProvider} for Iceberg tables, mirroring the paimon connector's @@ -495,12 +496,13 @@ public ConnectorSplitSource streamSplits(ConnectorSession session, ConnectorTabl boolean partitioned = table.spec().isPartitioned(); Map vendedToken = context != null ? extractVendedToken(table, restVendedCredentialsEnabled()) : Collections.emptyMap(); + UnaryOperator uriNormalizer = newUriNormalizer(vendedToken); long fileSplitSize = sessionLong(session, FILE_SPLIT_SIZE, 0L); long sliceSize = fileSplitSize > 0 ? fileSplitSize : sessionLong(session, MAX_FILE_SPLIT_SIZE, DEFAULT_MAX_FILE_SPLIT_SIZE); CloseableIterable tasks = streamingFileScanTasks(scan, session, table, filter, sliceSize); return new IcebergStreamingSplitSource(tasks, table, formatVersion, partitioned, - orderedPartitionKeys, zone, vendedToken, sliceSize, iceHandle.getRewriteFileScope()); + orderedPartitionKeys, zone, uriNormalizer, sliceSize, iceHandle.getRewriteFileScope()); } /** @@ -540,7 +542,7 @@ private final class IcebergStreamingSplitSource implements ConnectorSplitSource private final boolean partitioned; private final List orderedPartitionKeys; private final ZoneId zone; - private final Map vendedToken; + private final UnaryOperator uriNormalizer; private final long sliceSize; private final Set rewriteScope; // Lazily opened on first hasNext() so the ctor never throws — iceberg's ParallelIterable submits @@ -554,14 +556,14 @@ private final class IcebergStreamingSplitSource implements ConnectorSplitSource IcebergStreamingSplitSource(CloseableIterable tasks, Table table, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, - Map vendedToken, long sliceSize, Set rewriteScope) { + UnaryOperator uriNormalizer, long sliceSize, Set rewriteScope) { this.tasks = tasks; this.table = table; this.formatVersion = formatVersion; this.partitioned = partitioned; this.orderedPartitionKeys = orderedPartitionKeys; this.zone = zone; - this.vendedToken = vendedToken; + this.uriNormalizer = uriNormalizer; this.sliceSize = sliceSize; this.rewriteScope = rewriteScope; } @@ -576,7 +578,7 @@ public boolean hasNext() { } while (iterator.hasNext()) { IcebergScanRange range = buildRangeForTask(iterator.next(), table, formatVersion, partitioned, - orderedPartitionKeys, zone, vendedToken, sliceSize, rewriteScope, false, null); + orderedPartitionKeys, zone, uriNormalizer, sliceSize, rewriteScope, false, null); if (range != null) { buffered = range; return true; @@ -646,6 +648,9 @@ private List planScanInternal( // BE-credential overlay is emitted separately by getScanNodeProperties. Map vendedToken = context != null ? extractVendedToken(table, restVendedCredentialsEnabled()) : Collections.emptyMap(); + // Derive the vended storage config ONCE per scan (the token is scan-invariant) and reuse it for every + // per-file path normalization below, instead of rebuilding it per data/delete file (C3). + UnaryOperator uriNormalizer = newUriNormalizer(vendedToken); // COUNT(*) pushdown (T05): when the count is servable from the snapshot summary, collapse the scan to // a single whole-file range carrying the full count (mirrors paimon's collapse + legacy's <=10000 @@ -656,7 +661,7 @@ private List planScanInternal( long realCount = getCountFromSnapshot(scan, session); if (realCount >= 0) { return planCountPushdown(table, scan, realCount, formatVersion, partitioned, - orderedPartitionKeys, zone, vendedToken, session, filter); + orderedPartitionKeys, zone, uriNormalizer, session, filter); } } @@ -690,7 +695,7 @@ private List planScanInternal( // Shared per-task mapping (rewrite-scope skip + M-2 weight denominator + v3 stash side-effect), // identical to the streaming path's IcebergStreamingSplitSource so both produce the same ranges. IcebergScanRange range = buildRangeForTask(task, table, formatVersion, partitioned, - orderedPartitionKeys, zone, vendedToken, plan.targetSplitSize, rewriteScope, + orderedPartitionKeys, zone, uriNormalizer, plan.targetSplitSize, rewriteScope, stashRewritableDeletes, stashQueryId); if (range != null) { ranges.add(range); @@ -749,7 +754,7 @@ private static String rootCauseMessage(Throwable t) { */ private IcebergScanRange buildRangeForTask(FileScanTask task, Table table, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, - Map vendedToken, long targetSplitSize, Set rewriteScope, + UnaryOperator uriNormalizer, long targetSplitSize, Set rewriteScope, boolean stashRewritableDeletes, String stashQueryId) { DataFile dataFile = task.file(); if (rewriteScope != null && !rewriteScope.contains(dataFile.path().toString())) { @@ -758,7 +763,7 @@ private IcebergScanRange buildRangeForTask(FileScanTask task, Table table, int f // targetSplitSize is the scan-level weight denominator (M-2): each data-file range carries a // size-proportional BE scheduling weight (selfSplitWeight computed inside buildRange). IcebergScanRange range = buildRange(table, dataFile, task, formatVersion, partitioned, - orderedPartitionKeys, zone, vendedToken, -1, targetSplitSize); + orderedPartitionKeys, zone, uriNormalizer, -1, targetSplitSize); if (stashRewritableDeletes) { rewritableDeleteStash.accumulate(stashQueryId, range.getOriginalPath(), range.rewritableDeleteDescs()); @@ -934,12 +939,13 @@ private List doPlanPositionDeletesSystemTableScan(IcebergTab ZoneId zone = resolveSessionZone(session); Map vendedToken = context != null ? extractVendedToken(metadataTable, restVendedCredentialsEnabled()) : Collections.emptyMap(); + UnaryOperator uriNormalizer = newUriNormalizer(vendedToken); List ranges = new ArrayList<>(); for (PositionDeletesScanTask task : tasks) { for (PositionDeletesScanTask splitTask : splitPositionDeleteScanTask(task, targetSplitSize)) { ranges.add(buildPositionDeleteRange(splitTask, metadataTable, outputPartitionFields, - enableMappingVarbinary, zone, vendedToken)); + enableMappingVarbinary, zone, uriNormalizer)); } } LOG.debug("Iceberg planScan produced {} position_deletes splits for {}.{}", ranges.size(), @@ -959,11 +965,11 @@ private static Iterable splitPositionDeleteScanTask(Pos */ private IcebergScanRange buildPositionDeleteRange(PositionDeletesScanTask task, Table metadataTable, List outputPartitionFields, boolean enableMappingVarbinary, ZoneId zone, - Map vendedToken) { + UnaryOperator uriNormalizer) { DeleteFile deleteFile = task.file(); String originalPath = deleteFile.path().toString(); IcebergScanRange.Builder builder = new IcebergScanRange.Builder() - .path(normalizeUri(originalPath, vendedToken)) + .path(uriNormalizer.apply(originalPath)) .start(task.start()) .length(task.length()) .fileSize(deleteFile.fileSizeInBytes()) @@ -1151,13 +1157,13 @@ private static Schema pinnedSchema(Table table, IcebergTableHandle handle) { */ private List planCountPushdown(Table table, TableScan scan, long realCount, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, - Map vendedToken, ConnectorSession session, Optional filter) { + UnaryOperator uriNormalizer, ConnectorSession session, Optional filter) { try (CloseableIterable tasks = countPushdownFileScanTasks(scan, session, table, filter)) { for (FileScanTask task : tasks) { // targetSplitSize = -1: the count-pushdown collapse emits a single range, so its scheduling // weight is irrelevant → PluginDrivenSplit keeps SplitWeight.standard(). return Collections.singletonList(buildRange(table, task.file(), task, formatVersion, - partitioned, orderedPartitionKeys, zone, vendedToken, realCount, -1)); + partitioned, orderedPartitionKeys, zone, uriNormalizer, realCount, -1)); } } catch (IOException e) { throw new RuntimeException("Failed to plan iceberg count-pushdown file, error message is:" @@ -1200,7 +1206,7 @@ private CloseableIterable countPushdownFileScanTasks(TableScan sca */ private IcebergScanRange buildRange(Table table, DataFile dataFile, FileScanTask task, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, - Map vendedToken, long pushDownRowCount, long targetSplitSize) { + UnaryOperator uriNormalizer, long pushDownRowCount, long targetSplitSize) { Integer partitionSpecId = null; String partitionDataJson = null; Map partitionValues = Collections.emptyMap(); @@ -1258,7 +1264,7 @@ private IcebergScanRange buildRange(Table table, DataFile dataFile, FileScanTask // position-delete entries against the raw iceberg path (legacy setOriginalFilePath:304). String rawDataPath = dataFile.path().toString(); return new IcebergScanRange.Builder() - .path(normalizeUri(rawDataPath, vendedToken)) + .path(uriNormalizer.apply(rawDataPath)) .originalPath(rawDataPath) .start(task.start()) .length(task.length()) @@ -1270,7 +1276,7 @@ private IcebergScanRange buildRange(Table table, DataFile dataFile, FileScanTask .firstRowId(firstRowId) .lastUpdatedSequenceNumber(lastUpdatedSequenceNumber) .partitionValues(partitionValues) - .deleteFiles(buildDeleteFiles(task, vendedToken)) + .deleteFiles(buildDeleteFiles(task, uriNormalizer)) .pushDownRowCount(pushDownRowCount) .selfSplitWeight(selfSplitWeight) .targetSplitSize(targetSplitSize) @@ -1282,14 +1288,15 @@ private IcebergScanRange buildRange(Table table, DataFile dataFile, FileScanTask * mirroring legacy {@code IcebergScanNode.getDeleteFileFilters} + {@code IcebergDeleteFileFilter}. Empty * for v1 / no-delete files (v1 has no delete files, so {@code task.deletes()} is always empty there). */ - private List buildDeleteFiles(FileScanTask task, Map vendedToken) { + private List buildDeleteFiles(FileScanTask task, + UnaryOperator uriNormalizer) { List deletes = task.deletes(); if (deletes == null || deletes.isEmpty()) { return Collections.emptyList(); } List result = new ArrayList<>(deletes.size()); for (DeleteFile delete : deletes) { - result.add(convertDelete(delete, vendedToken)); + result.add(convertDelete(delete, uriNormalizer)); } return result; } @@ -1304,13 +1311,13 @@ private List buildDeleteFiles(FileScanTask task, Ma *

  • {@code EQUALITY_DELETES} → an equality delete (content 2) with the delete-file's equality * field-ids (read straight from delete metadata — correct independent of the T06 data dictionary).
    • * - * The delete path is normalized through the engine seam (legacy - * {@code LocationPath.of(path,config).toStorageLocation()}), threading the per-table vended token (empty - * for non-REST) so a REST object-store deletion path normalizes via the vended map (T09). Package-private - * for direct unit testing. + * The delete path is normalized through the scan-scoped {@code uriNormalizer} (legacy + * {@code LocationPath.of(path,config).toStorageLocation()}), which bakes in the per-table vended token + * (empty for non-REST) so a REST object-store deletion path normalizes via the vended map (T09). + * Package-private for direct unit testing. */ - IcebergScanRange.DeleteFile convertDelete(DeleteFile delete, Map vendedToken) { - String path = normalizeUri(delete.path().toString(), vendedToken); + IcebergScanRange.DeleteFile convertDelete(DeleteFile delete, UnaryOperator uriNormalizer) { + String path = uriNormalizer.apply(delete.path().toString()); FileContent content = delete.content(); if (content == FileContent.POSITION_DELETES) { Long lowerBound = readPositionBound(delete.lowerBounds()); @@ -1367,20 +1374,22 @@ private static TFileFormatType deleteFileFormat(FileFormat format) { } /** - * Normalize a raw iceberg storage path (the data file BE opens, or a delete file) to BE's canonical - * scheme via the engine seam (legacy goes through {@code LocationPath.of(path, storagePropertiesMap) - * .toStorageLocation()}; the connector cannot import fe-core's {@code LocationPath}). BE's - * scheme-dispatched S3 factory only opens {@code s3://}, so an un-normalized {@code oss://}/{@code cos://} - * /{@code obs://}/{@code s3a://} path fails the native read (data file) or silently drops the deletes + * Build the scan-scoped URI normalizer once (where the per-table vended token is extracted) and thread it + * through the per-file range builders, instead of re-deriving the vended storage config per data/delete + * file. Each application normalizes a raw iceberg storage path (the data file BE opens, or a delete file) + * to BE's canonical scheme via the engine seam (legacy goes through {@code LocationPath.of(path, + * storagePropertiesMap).toStorageLocation()}; the connector cannot import fe-core's {@code LocationPath}). + * BE's scheme-dispatched S3 factory only opens {@code s3://}, so an un-normalized {@code oss://}/{@code + * cos://}/{@code obs://}/{@code s3a://} path fails the native read (data file) or silently drops the deletes * (merge-on-read wrong rows). Mirrors paimon's {@code normalizeUri} (FIX-URI-NORMALIZE), which normalizes * both the data-file and deletion-vector paths. The {@code vendedToken} (empty for non-REST / no context) - * is the per-table vended credential map, routed into normalization so a REST object-store path normalizes - * via the vended map (T09); when empty the 2-arg seam folds to the catalog's static storage map, byte- - * equivalent to legacy for non-vended catalogs. A {@code null} context (offline unit tests) preserves the - * raw path (paimon parity). + * is the per-table vended credential map, baked into the normalizer so a REST object-store path normalizes + * via the vended map (T09); when empty the seam folds to the catalog's static storage map, byte-equivalent + * to legacy for non-vended catalogs. A {@code null} context (offline unit tests) yields an identity + * normalizer that preserves the raw path (paimon parity). */ - private String normalizeUri(String rawPath, Map vendedToken) { - return context != null ? context.normalizeStorageUri(rawPath, vendedToken) : rawPath; + UnaryOperator newUriNormalizer(Map vendedToken) { + return context != null ? context.newStorageUriNormalizer(vendedToken) : UnaryOperator.identity(); } /** diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java index e6e70530cfef0a..380f77ec65b915 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java @@ -30,6 +30,7 @@ import java.util.Objects; import java.util.concurrent.Callable; import java.util.function.Supplier; +import java.util.function.UnaryOperator; /** * A {@link ConnectorContext} decorator that pins the thread-context classloader (TCCL) to the iceberg plugin @@ -167,6 +168,15 @@ public String normalizeStorageUri(String rawUri, Map rawVendedCr return delegate.normalizeStorageUri(rawUri, rawVendedCredentials); } + @Override + public UnaryOperator newStorageUriNormalizer(Map rawVendedCredentials) { + // Delegate to the raw engine context so the connector gets DefaultConnectorContext's once-per-scan + // hoist. Without this override the SPI default would fold back to THIS wrapper's per-call + // normalizeStorageUri, silently defeating the optimization. Like normalizeStorageUri, this path runs + // entirely in fe-core (LocationPath/StorageProperties, no plugin reflection), so no TCCL pin is needed. + return delegate.newStorageUriNormalizer(rawVendedCredentials); + } + @Override public String getBackendFileType(String rawUri, Map rawVendedCredentials) { return delegate.getBackendFileType(rawUri, rawVendedCredentials); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index 41c038b0780c86..56d5e2b6af7e40 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -83,6 +83,7 @@ import java.util.NoSuchElementException; import java.util.Optional; import java.util.Set; +import java.util.function.UnaryOperator; /** * Tests for {@link IcebergScanPlanProvider}. T01 pinned the capability constants + that {@code planScan} @@ -1559,7 +1560,7 @@ public void convertDeletePositionDeleteCarriesBoundsAndFormat() { // POSITION_DELETES (non-PUFFIN) -> content 1, parquet/orc format, [lower,upper] bounds decoded from the // delete file's DELETE_FILE_POS bounds. MUTATION: wrong content id / dropped bounds / wrong format -> red. DeleteFile delete = positionDeleteFile("s3://b/db/t1/pos.parquet", FileFormat.PARQUET, 3L, 17L); - TIcebergDeleteFileDesc d = provider().convertDelete(delete, Collections.emptyMap()).toThrift(); + TIcebergDeleteFileDesc d = provider().convertDelete(delete, UnaryOperator.identity()).toThrift(); Assertions.assertEquals(1, d.getContent()); Assertions.assertEquals("s3://b/db/t1/pos.parquet", d.getPath()); @@ -1575,7 +1576,7 @@ public void convertDeletePositionDeleteWithoutBoundsLeavesThemUnset() { // No DELETE_FILE_POS bounds present -> position_lower/upper_bound stay unset (legacy emits them only // when present; it stores a -1 sentinel and skips emission). MUTATION: emitting 0/-1 -> red. DeleteFile delete = positionDeleteFile("s3://b/db/t1/pos.orc", FileFormat.ORC, null, null); - TIcebergDeleteFileDesc d = provider().convertDelete(delete, Collections.emptyMap()).toThrift(); + TIcebergDeleteFileDesc d = provider().convertDelete(delete, UnaryOperator.identity()).toThrift(); Assertions.assertEquals(TFileFormatType.FORMAT_ORC, d.getFileFormat()); Assertions.assertFalse(d.isSetPositionLowerBound()); @@ -1588,7 +1589,7 @@ public void convertDeleteDeletionVectorCarriesBlobRefAndUnsetsFormat() { // (legacy setDeleteFileFormat skips PUFFIN). MUTATION: classifying it as content 1 / emitting a format // for the puffin blob -> red (BE would mis-read the DV blob). DeleteFile delete = deletionVectorFile("s3://b/db/t1/dv.puffin", 16L, 64L); - TIcebergDeleteFileDesc d = provider().convertDelete(delete, Collections.emptyMap()).toThrift(); + TIcebergDeleteFileDesc d = provider().convertDelete(delete, UnaryOperator.identity()).toThrift(); Assertions.assertEquals(3, d.getContent()); Assertions.assertFalse(d.isSetFileFormat()); @@ -1663,7 +1664,7 @@ public void convertDeleteEqualityDeleteCarriesFieldIds() { // the T06 data-schema dictionary). MUTATION: wrong content id / dropped field-ids -> red (BE projects // the wrong columns for the equality match). DeleteFile delete = equalityDeleteFile("s3://b/db/t1/eq.parquet", FileFormat.PARQUET, 1, 2); - TIcebergDeleteFileDesc d = provider().convertDelete(delete, Collections.emptyMap()).toThrift(); + TIcebergDeleteFileDesc d = provider().convertDelete(delete, UnaryOperator.identity()).toThrift(); Assertions.assertEquals(2, d.getContent()); Assertions.assertEquals(Arrays.asList(1, 2), d.getFieldIds()); @@ -1683,7 +1684,8 @@ public void convertDeleteNormalizesDeletePathViaContext() { new IcebergScanPlanProvider(Collections.emptyMap(), new RecordingIcebergCatalogOps(), context); DeleteFile delete = positionDeleteFile("oss://bucket/db/t1/pos.parquet", FileFormat.PARQUET, null, null); - TIcebergDeleteFileDesc d = provider.convertDelete(delete, Collections.emptyMap()).toThrift(); + TIcebergDeleteFileDesc d = + provider.convertDelete(delete, provider.newUriNormalizer(Collections.emptyMap())).toThrift(); Assertions.assertEquals("s3://bucket/db/t1/pos.parquet", d.getPath()); Assertions.assertTrue(context.normalizedUris.contains("oss://bucket/db/t1/pos.parquet")); @@ -2410,6 +2412,32 @@ public void planScanThreadsVendedTokenIntoDataAndDeletePathNormalize() { Assertions.assertEquals(IcebergScanPlanProvider.extractVendedToken(table, false), context.lastVendedToken); } + @Test + public void planScanDerivesUriNormalizerOncePerScanNotPerFile() { + // C3 PERF GUARD: the vended token is scan-invariant, so the expensive token->storage-config + // derivation must be built ONCE per scan and reused for every file path — not rebuilt per data file. + // Drive a scan over three data files and assert the connector entered newStorageUriNormalizer exactly + // once while still normalizing all three paths. MUTATION: reverting to a per-file + // context.normalizeStorageUri (re-deriving the config per file) leaves newNormalizerCount == 0 (the + // once-per-scan seam is never used) -> red; dropping a path's normalize -> normalizeCount != 3 -> red. + Table table = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + table.newAppend() + .appendFile(dataFile(table.spec(), "oss://b/db/t1/f1.parquet", 1024, null, null)) + .appendFile(dataFile(table.spec(), "oss://b/db/t1/f2.parquet", 1024, null, null)) + .appendFile(dataFile(table.spec(), "oss://b/db/t1/f3.parquet", 1024, null, null)) + .commit(); + RecordingConnectorContext context = new RecordingConnectorContext(); + IcebergScanPlanProvider provider = + new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table), context); + + List ranges = provider.planScan( + null, new IcebergTableHandle("db1", "t1"), Collections.emptyList(), Optional.empty()); + + Assertions.assertEquals(3, ranges.size()); + Assertions.assertEquals(1, context.newNormalizerCount); + Assertions.assertEquals(3, context.normalizeCount); + } + @Test public void convertDeleteNormalizesDeletePathViaVendedToken() { RecordingConnectorContext context = new RecordingConnectorContext(); @@ -2418,10 +2446,10 @@ public void convertDeleteNormalizesDeletePathViaVendedToken() { DeleteFile delete = positionDeleteFile("oss://bucket/db/t1/pos.parquet", FileFormat.PARQUET, null, null); Map token = Collections.singletonMap("s3.access-key-id", "ak"); - TIcebergDeleteFileDesc d = provider.convertDelete(delete, token).toThrift(); + TIcebergDeleteFileDesc d = provider.convertDelete(delete, provider.newUriNormalizer(token)).toThrift(); - // WHY: convertDelete must thread the vended token into the 2-arg normalize (T09). MUTATION: passing no - // token / the 1-arg normalize -> lastVendedToken != token -> red. + // WHY: the scan-scoped normalizer must bake in the vended token so the delete path normalizes via the + // 2-arg seam (T09). MUTATION: dropping the token / the 1-arg normalize -> lastVendedToken != token -> red. Assertions.assertEquals("s3://bucket/db/t1/pos.parquet", d.getPath()); Assertions.assertEquals(token, context.lastVendedToken); } @@ -2509,14 +2537,14 @@ public void convertDeletePositionDeleteTreatsStoredMinusOneBoundAsUnset() { // sentinel. The existing no-bounds test passes a null map (early return), never reaching the value==-1L // arm. MUTATION: dropping the `|| value == -1L` arm (emitting -1 as a real bound) -> red. DeleteFile bothMinusOne = positionDeleteFile("s3://b/db/t1/pos.parquet", FileFormat.PARQUET, -1L, -1L); - TIcebergDeleteFileDesc d = provider().convertDelete(bothMinusOne, Collections.emptyMap()).toThrift(); + TIcebergDeleteFileDesc d = provider().convertDelete(bothMinusOne, UnaryOperator.identity()).toThrift(); Assertions.assertEquals(1, d.getContent()); Assertions.assertFalse(d.isSetPositionLowerBound()); Assertions.assertFalse(d.isSetPositionUpperBound()); // Mixed: only the -1L bound is dropped; a real lower bound still emits. DeleteFile mixed = positionDeleteFile("s3://b/db/t1/pos2.parquet", FileFormat.PARQUET, 3L, -1L); - TIcebergDeleteFileDesc m = provider().convertDelete(mixed, Collections.emptyMap()).toThrift(); + TIcebergDeleteFileDesc m = provider().convertDelete(mixed, UnaryOperator.identity()).toThrift(); Assertions.assertEquals(3L, m.getPositionLowerBound()); Assertions.assertFalse(m.isSetPositionUpperBound()); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingConnectorContext.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingConnectorContext.java index 8956a899630320..7876783a04d6c2 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingConnectorContext.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingConnectorContext.java @@ -29,6 +29,7 @@ import java.util.List; import java.util.Map; import java.util.concurrent.Callable; +import java.util.function.UnaryOperator; /** * Hand-written {@link ConnectorContext} test double (no Mockito), adapted verbatim from the paimon @@ -60,6 +61,9 @@ final class RecordingConnectorContext implements ConnectorContext { final List normalizedUris = new ArrayList<>(); /** Number of times the connector invoked {@link #normalizeStorageUri} (1- or 2-arg). */ int normalizeCount; + /** Number of times the connector built a scan-scoped normalizer via {@link #newStorageUriNormalizer} + * (should be once per scan — the perf hoist guard). */ + int newNormalizerCount; /** The vended token the connector passed to the most recent 2-arg {@link #normalizeStorageUri} (T09). */ Map lastVendedToken; @@ -119,6 +123,16 @@ public String normalizeStorageUri(String rawUri, Map vendedToken return rawUri == null ? null : rawUri.replaceFirst("^(oss|cos|obs|s3a)://", "s3://"); } + @Override + public UnaryOperator newStorageUriNormalizer(Map vendedToken) { + // Count the once-per-scan derivation (the perf hoist) but still record each per-URI normalize by + // delegating every apply back to the recording normalizeStorageUri — so existing recording assertions + // (normalizedUris / normalizeCount / lastVendedToken) keep firing, while newNormalizerCount proves the + // token->config derivation is entered once per scan, not once per file. + newNormalizerCount++; + return rawUri -> normalizeStorageUri(rawUri, vendedToken); + } + @Override public List getStorageProperties() { return storageProperties; diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java index 5545697e12708a..c38c8a9c91044e 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java @@ -27,6 +27,7 @@ import java.util.List; import java.util.Map; import java.util.concurrent.Callable; +import java.util.function.UnaryOperator; /** * Runtime context provided by fe-core to connector implementations. @@ -208,6 +209,28 @@ default String normalizeStorageUri(String rawUri, Map rawVendedC return normalizeStorageUri(rawUri); } + /** + * Scan-scoped batch form of {@link #normalizeStorageUri(String, Map)}: derives the vended storage + * configuration from the (scan-invariant) per-table token ONCE and returns a normalizer that applies + * it to many raw URIs cheaply. A vended-credentials scan normalizes O(N_files + N_deletes) paths but + * the token→storage-config derivation ({@code StorageProperties.createAll} + a hadoop config build) is + * a pure function of the token, so hoisting it out of the per-file loop turns O(N) heavy derivations + * into one. The connector builds the normalizer once (where it extracts the token) and reuses it for + * every data/delete/position-delete path in the scan. + * + *

    The default returns a normalizer that delegates per call to {@link #normalizeStorageUri(String, + * Map)} — behavior-identical, no hoist — so a connector with no engine context (offline unit tests) + * and any connector that does not override the engine side are unaffected. The engine + * ({@code DefaultConnectorContext}) overrides this to perform the actual once-per-scan derivation. + * + * @param rawVendedCredentials the raw per-table vended token map (may be null/empty → static path) + * @return a URI normalizer for this scan; each application is byte-identical to + * {@link #normalizeStorageUri(String, Map)} with the same token + */ + default UnaryOperator newStorageUriNormalizer(Map rawVendedCredentials) { + return rawUri -> normalizeStorageUri(rawUri, rawVendedCredentials); + } + /** * Resolves the BE-facing file type (a {@code TFileType} enum name, e.g. {@code "FILE_S3"}) for a raw * storage URI a connector emits (e.g. an iceberg write output path). A write-side analogue of diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java index 8aa89490f55321..522e3782517787 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java @@ -66,6 +66,7 @@ import java.util.concurrent.Callable; import java.util.function.Function; import java.util.function.Supplier; +import java.util.function.UnaryOperator; import java.util.stream.Collectors; /** @@ -408,6 +409,37 @@ public String normalizeStorageUri(String rawUri, Map rawVendedCr return LocationPath.of(rawUri, effective).toStorageLocation().toString(); } + @Override + public UnaryOperator newStorageUriNormalizer(Map rawVendedCredentials) { + // PERF: the vended token is scan-invariant, so derive the effective storage map (the expensive + // buildVendedStorageMap = StorageProperties.createAll + hadoop config build) ONCE per scan and reuse + // it for every per-file normalize, instead of rebuilding it per data/delete file. Each application is + // byte-identical to normalizeStorageUri(rawUri, token): the SAME empty-uri short-circuit, the SAME + // vended-replaces-static precedence, the SAME fail-loud LocationPath. The derivation is done LAZILY on + // the first non-empty URI (not eagerly at construction) so a scan that normalizes zero non-empty URIs + // triggers no derivation — preserving the exact exception timing of the per-call method. The returned + // normalizer is single-threaded per scan (the streaming pump drives one thread; the synchronous and + // position-delete loops are single-threaded), so the memo needs no lock. + return new UnaryOperator() { + private Map effective; + private boolean built; + + @Override + public String apply(String rawUri) { + if (Strings.isNullOrEmpty(rawUri)) { + return rawUri; + } + if (!built) { + Map vended = + buildVendedStorageMap(rawVendedCredentials); + effective = vended != null ? vended : storagePropertiesSupplier.get(); + built = true; + } + return LocationPath.of(rawUri, effective).toStorageLocation().toString(); + } + }; + } + @Override public String getBackendFileType(String rawUri, Map rawVendedCredentials) { // Same LocationPath build as normalizeStorageUri (vended-aware), then read the BE file type from diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextNormalizeUriTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextNormalizeUriTest.java index ff22c40f949f92..b2f95c634b8fb8 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextNormalizeUriTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextNormalizeUriTest.java @@ -30,6 +30,7 @@ import java.util.Map; import java.util.function.Function; import java.util.function.Supplier; +import java.util.function.UnaryOperator; import java.util.stream.Collectors; /** @@ -169,4 +170,57 @@ public void backendFileTypeVendedRestResolvesUnderEmptyStaticMap() { Assertions.assertEquals(TFileType.FILE_S3.name(), restCtx.getBackendFileType("oss://bkt/warehouse/db/t/data", ossVendedToken())); } + + // ---- FIX-PERF-06: newStorageUriNormalizer hoists the (scan-invariant) token->storage-config + // derivation to ONCE per scan; every application must stay byte-identical to a per-call + // normalizeStorageUri(uri, token), across all four cases the per-call form covers. ---- + + @Test + public void newNormalizerVendedMatchesPerCallAndServesManyUris() { + // WHY: the scan-scoped normalizer bakes the vended token in once, then normalizes many paths; + // each application must equal normalizeStorageUri(uri, token) (REST empty-static -> vended + // replaces static), and ONE normalizer must serve multiple files (the whole point of the hoist). + // MUTATION: dropping the token (static-only) throws; a stale/rebuilt map yielding a different path + // -> red. + DefaultConnectorContext restCtx = new DefaultConnectorContext("c", 1L); + UnaryOperator n = restCtx.newStorageUriNormalizer(ossVendedToken()); + Assertions.assertEquals(restCtx.normalizeStorageUri("oss://bkt/a/f1.parquet", ossVendedToken()), + n.apply("oss://bkt/a/f1.parquet")); + Assertions.assertEquals("s3://bkt/a/f1.parquet", n.apply("oss://bkt/a/f1.parquet")); + // Reuse the SAME normalizer for a second, different path — one derivation, many applications. + Assertions.assertEquals("s3://bkt/b/f2.parquet", n.apply("oss://bkt/b/f2.parquet")); + } + + @Test + public void newNormalizerStaticMapMatchesPerCallUnderEmptyToken() throws Exception { + // WHY: with a static OSS map and an empty token, the normalizer folds to the static-map path, + // byte-identical to the per-call form. MUTATION: an empty token suppressing the static map -> red. + DefaultConnectorContext ctx = ossContext(); + UnaryOperator n = ctx.newStorageUriNormalizer(Collections.emptyMap()); + Assertions.assertEquals( + ctx.normalizeStorageUri("oss://bkt/warehouse/db/t/part-0.parquet", Collections.emptyMap()), + n.apply("oss://bkt/warehouse/db/t/part-0.parquet")); + } + + @Test + public void newNormalizerShortCircuitsNullAndBlankWithoutForcingDerivation() throws Exception { + // WHY: same empty-uri short-circuit as normalizeStorageUri — a null/blank path returns unchanged + // and never reaches the fail-loud LocationPath, even on an empty static map + empty token (so a + // scan that only ever sees blank uris triggers no derivation/throw). MUTATION: NPE / fabricated + // output / forcing the derivation to throw -> red. + DefaultConnectorContext restCtx = new DefaultConnectorContext("c", 1L); + UnaryOperator n = restCtx.newStorageUriNormalizer(Collections.emptyMap()); + Assertions.assertNull(n.apply(null)); + Assertions.assertEquals("", n.apply("")); + } + + @Test + public void newNormalizerFailsLoudOnBadPathLikePerCall() { + // WHY: fail-loud parity — an empty static map + empty token has no credential, so applying to a + // real oss:// path must throw (not ship the raw path to BE), exactly like normalizeStorageUri. + // MUTATION: swallowing to the raw path -> red. + DefaultConnectorContext restCtx = new DefaultConnectorContext("c", 1L); + UnaryOperator n = restCtx.newStorageUriNormalizer(Collections.emptyMap()); + Assertions.assertThrows(RuntimeException.class, () -> n.apply("oss://bkt/a/part-0.parquet")); + } } From fdbeede298004332d0c6608ac4f5391035c27388 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 11:55:00 +0800 Subject: [PATCH 15/47] [doc](catalog) fe-connector-iceberg PERF-06: design + summary + tracker updates; next = PERF-07 Record the route-A decision (connector-side "prepare once, apply many" via a new scan-scoped SPI normalizer seam, rejecting the fe-core per-catalog memo for its cross-query thrash + credential-retention risk), the adversarial red-team outcome (lazy-memoized effective for exact exception-timing parity), and the two-stage build verification (fe-core must be built separately since the iceberg connector does not depend on it). Roll the tasklist/progress/HANDOFF forward to PERF-07 (C20 write-path 3-5 loads of the same table). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 54 ++++----- ...F-06-vended-uri-normalizer-hoist-design.md | 106 ++++++++++++++++++ ...-06-vended-uri-normalizer-hoist-summary.md | 34 ++++++ plan-doc/perf-hotpath-iceberg/progress.md | 14 +++ plan-doc/perf-hotpath-iceberg/tasklist.md | 10 +- 5 files changed, 188 insertions(+), 30 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-06-vended-uri-normalizer-hoist-design.md create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-06-vended-uri-normalizer-hoist-summary.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index 7eb04a68b5364c..f446c19a0e066e 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -6,55 +6,57 @@ --- -# ✅ 已完成（PERF-01 ~ 05） +# ✅ 已完成（PERF-01 ~ 06） - **PERF-01**（`484f0e0c125`）：胖 handle + 跨查询 `IcebergTableCache`（**凭证 gate**）。规划期 loadTable 3~7→1。 - **PERF-02**（`518d0599cbf`）：`IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、无 gate）。分区扫描每查询→每快照。 - **PERF-03**（`0b96f2e6c78`）：`IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、无 gate）。格式推断每查询→每快照。 - **PERF-04**（`2e5f393779c`）：抽惰性 `cacheBackedFileScanTasks` 三处复用，manifest cache 接回大表流式 + COUNT(*)（缓存与不 OOM 兼得）。 -- **PERF-05**（`aea3ebdd40e`）：`IcebergCommentCache`（键 `TableIdentifier`、无 gate），**仅 vended 非 session=user**（红队：session=user 缓存绕过 per-user 授权=泄漏）。information_schema comment 每表 load 跨查询坍缩（vended）。小结 `designs/FIX-PERF-05-*-summary.md`。 -- 五轮全 iceberg 模块单测绿（→973 / 0 fail / 1 skip），checkstyle 绿，0 回归。 -- **可复用产物**：四套缓存基建（table/partition/format/comment）；**gate 三判据**：①值含 FileIO/凭证→gate；②纯元数据→无 gate；③**授权发生在 load 调用里（session=user）→即便值无凭证也不可共享缓存**（PERF-05 新增，与①②正交叠加）。「缓存只存 loader 原始输出、失败透传不缓存」；「delete 索引 eager + 惰性扁平映射」（PERF-04）。 +- **PERF-05**（`aea3ebdd40e`）：`IcebergCommentCache`（键 `TableIdentifier`、无 gate），**仅 vended 非 session=user**。information_schema comment 每表 load 跨查询坍缩。 +- **PERF-06**（`6294edf2833`）：vended-credentials URI 归一化 **scan 级「准备一次、逐文件套用」**。新增 SPI `newStorageUriNormalizer`（默认逐文件折回、零影响其它连接器），`DefaultConnectorContext` override 惰性 memo 一次派生；iceberg 三 scan-start 各构造一次 normalizer 穿六 seam；`TcclPinning` 透传。**否决 fe-core 框架 memo**（per-catalog 跨查询共享→并发冲刷+凭证保留）。一次 vended scan `StorageProperties.createAll` O(N_files+N_deletes)→1。小结 `designs/FIX-PERF-06-*-summary.md`。 +- iceberg 全模块单测绿（→974 / 0 fail / 1 skip），fe-core `DefaultConnectorContextNormalizeUriTest` 13 绿，checkstyle 绿，0 回归。 +- **可复用产物**：四套缓存基建（table/partition/format/comment）+ 一个 SPI「准备一次、套用多次」normalizer 范式；**gate 判据三条**（①值含凭证→gate ②纯元数据→无 gate ③授权在 load 里/session=user→不可共享）；**PERF-06 新判据「作用域即安全边界」**：同一份含凭证派生，跨查询做撞 gate、**scan 级做天然安全**（token 恒定、无跨用户、结束即回收）。 --- -# 🆕 下一个 session = **PERF-06（C3 REST vended-credentials 每文件重建 StorageProperties+Configuration）** +# 🆕 下一个 session = **PERF-07（C20 写路径一条 DML 3~5 次 load 同表）** -## 病灶 +## 病灶（审计基线行号，动码前必重 grep） -REST vended-credentials 目录，扫描规划期 `buildRange → normalizeUri → context.normalizeStorageUri(rawUri, vendedToken)`（`TcclPinningConnectorContext` → **fe-core** `DefaultConnectorContext.normalizeStorageUri:392-409`）→ `buildVendedStorageMap:225-242`（`CredentialUtils.filterCloudStorageProperties` + `StorageProperties.createAll`：遍历所有 provider + **建 hadoop `Configuration` + 逐 key set**）。**每 data-file range 一次 + 每 MOR delete file per task 一次 + 每 position-delete split 一次**（同步/流式/$position_deletes 三路径同）。vendedToken **整个 scan 内不变**（`planScanInternal:586-587` sync / `:458-459` streaming 只提取一次、同一 Map 实例穿所有调用），但每文件都重建整份 map+Configuration。50k 文件 ≈ **数十秒纯 FE CPU**（MOR 加倍）。门槛：REST + `iceberg.rest.vended-credentials-enabled=true`（空 token 在 `:227-229` 短路）。 +一条 iceberg DML（INSERT/DELETE/MERGE）在规划+绑定期对同一张表串行 load 3~5 次： +`PhysicalIcebergMergeSink.getRequirePhysicalProperties:161→198`、`PhysicalPlanTranslator.visitPhysicalConnectorTableSink:675,703`、`PluginDrivenTableSink.bindDataSink:175` → `IcebergWritePlanProvider.resolveTable:689-702` / `beginWrite`（内含 `tableExists` ×2 + **无条件 refresh**）。每条 DML +3~5 次串行远程往返。 +**修复方向（审计）**：语句级 resolve 一次传递；`exists` 从 load 结果推导（load 成功即存在，不必单独 tableExists）。 ## 第一件事（立项流程见 README §单项立项流程） -1. **复核（动码前，行号信 grep；PERF-01~05 后已漂移）**：grep `buildRange`/`convertDelete`/`buildPositionDeleteRange` 的 `normalizeUri` 调用点、`DefaultConnectorContext.normalizeStorageUri` + `buildVendedStorageMap`。**核心确认**：`buildVendedStorageMap(token)` 的重活（`StorageProperties.createAll` + hadoop Configuration）是**纯 token 派生、与 rawUri 无关**吗？（审计说是——URI 只用于路径归一化那半，cheap。） -2. **⚠ 路由决策（关键，可能需上交用户，像 PERF-04/05）**：`DefaultConnectorContext` 在 **fe-core**。两条路线： - - **(a) 连接器侧 hoist**：iceberg 扫描期把 token→map 派生**提升到 scan 级**（token 已只提取一次），per-file 只做 cheap 路径归一化。**难点**：`normalizeStorageUri(uri, token)` 是 fe-core `ConnectorContext` API，重活封装在 fe-core 里；连接器要 hoist 需 fe-core 暴露"派生一次 map"的入口，或连接器自己做路径归一化——**核 `ConnectorContext` API 表面**能否连接器侧 hoist 而不改 fe-core。 - - **(b) fe-core 框架 memo**：`DefaultConnectorContext` 内按 **token 恒等键单条目 memo** `buildVendedStorageMap`。这是 fe-core **框架层**改动——按 README 铁律 2（PERF-09 先例：**改通用框架、惠及所有连接器、非 source-specific = 允许**）**大概率合规**，但属**共享热路径**，须证 **byte + cost 对所有连接器双不变**（对齐 PERF-09/共享 MVCC 纪律），且 token 作 key 要用**恒等/内容**键谨慎（同 Map 实例→identity 键最稳）。 - - **倾向**：先核 (a) 可行性；若 (a) 须改 fe-core API 才能 hoist，则 (b)（框架 memo，惠及所有 vended 连接器）反而更干净——但**两条都动到 fe-core 边界**，故**动码前把选择+理由用中文上交用户定**（不引用任务代号）。 -3. **红队 + TDD**：度量守门 = 一次 vended scan 内 `buildVendedStorageMap`/`StorageProperties.createAll` 远端/CPU 次数从 O(N_files+N_deletes)→1（可加计数器或 spy）。 -4. 守铁律：优先连接器侧；若走 fe-core 框架 memo，须 connector-agnostic（禁按源名分支）+ 双不变证明。 +1. **复核（动码前，行号信 grep；PERF-01~06 后已漂移）**：grep 上述五处 load/resolve/beginWrite/tableExists 调用点，重新确认：①这些 load 是否已被 PERF-01 的 `IcebergTableCache` 部分兜住（写路径是否走 resolveTableForRead？还是另一条 write-only 的 resolve/refresh？**PERF-05 复核先例：审计早于 PERF-01，残余可能已缩小**）②`beginWrite` 的**无条件 refresh** 是真必需还是可省（refresh 破坏 PERF-01 缓存命中？）③`tableExists ×2` 能否从 load 结果推导。 +2. **⚠ 路由/归属**：写路径 resolve 分散在 **Nereids 物理算子层（fe-core `PhysicalPlanTranslator`/`PluginDrivenTableSink`）** 与 **连接器 `IcebergWritePlanProvider`** 两侧。语句级"resolve 一次传递"若要跨这两层，可能须动 fe-core 的 sink 绑定流程——**先核能否纯连接器侧收敛**（`IcebergWritePlanProvider` 内 statement 级 memo，键=queryId/statementId + TableIdentifier），若须动 fe-core 通用 sink 流程则**动码前上交用户**（对齐 PERF-06/铁律）。 +3. **红队 + TDD**：度量守门 = 一条 DML 内对同表的 loadTable 远端次数从 3~5→1（可加计数器/spy，镜像 PERF-01 的 `loadCountForTest`）。**行为 parity 是硬闸门**：写语义（exists 判定、refresh 拿到最新快照）不得变——若省 refresh，须证写入仍绑定到正确的最新 metadata。 +4. 守铁律：优先连接器侧；exists 从 load 推导别新增 RPC；别把 Table 缓存塞进 fe-core。 ## ⚠️ 关键认知 -- **token loop-invariant 是本修的根**：token 每 scan 提取一次、同一 Map 实例穿所有 `normalizeUri`；重活是 token 派生、与 per-file URI 无关 → 可 memo/hoist 一次。 -- **三路径都要覆盖**：data-file range（buildRange）+ MOR delete（convertDelete）+ position-delete split（buildPositionDeleteRange），同步与流式两条。别只修一路。 -- **gate 判据延伸**（PERF-05 新增）：本任务缓存的是 storage map（**含凭证/token 派生**）→ 若做跨 scan 缓存会撞凭证 gate；但本修是 **scan 内**复用（token 恒定），非跨查询，故**不涉凭证泄漏**——scan 内 hoist/memo 安全。 +- **写路径 refresh 与 PERF-01 读缓存的张力**：PERF-01 的 `IcebergTableCache` 按 `(TableIdentifier, snapshotId)` 缓存；写路径若无条件 `refresh()` 会绕过/失效它。核清"写必须看最新"与"缓存命中"能否兼得（可能写路径本就该 pin 一次最新快照后全语句复用，类似 PERF-01/02 的 pin 模式）。 +- **statement 级作用域**：与 PERF-06 的 scan 级同理——写路径的 load 复用作用域是**一条语句**（queryId/statementId 稳定），非跨查询，故不涉凭证/授权 gate；memo 挂 statement context 或连接器侧按 queryId 键。 +- **exists 从 load 推导**：`tableExists` 单独 RPC 多半可删——load 成功即存在、load 抛 NoSuchTable 即不存在；但要核**建表/CTAS 分支**（表尚不存在时 exists=false 是正常路径，不能把"不存在"当错误抛）。 --- # 🧰 构建/验证坑（**实证，务必照做**） -1. **可靠跑单测 = `mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`**（绝对 `-f /fe/pom.xml`）。原因：`${revision}` + 未 flatten 已装 pom → `-pl iceberg` 解析不到 `fe-connector:pom:${revision}`，必须 `-am`；`-am test` 只到 test 相不产 hms-hive-shade shade jar（缺 `HiveConf`），故 **`install`**。`-Dtest=<类列表>`（`ls src/test/.../*Test.java` 拼逗号，本轮 56 类，存 `scratchpad/iceberg-tests.txt`）让上游快跳。 -2. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 静默跳过）。 -3. **别用 `mvn -q` 跑测试**（本轮踩坑）：成功时抑制 `BUILD SUCCESS`/`Tests run` INFO 行 → 无法从日志确认。**去掉 `-q`**，或用 surefire XML 聚合（`target/surefire-reports/TEST-*.xml` 求和 tests/failures/errors）验证。 -4. **别 `nohup ... &` 套 `run_in_background`**——外层立即随 echo 退出、maven 变孤儿，"exit 0" 假信号。直接 `mvn ... > log 2>&1`（run_in_background）。 -5. **checkstyle 主源 ≤120（test 源 suppress）**：本轮 2 处 javadoc 超行踩坑。import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行；**新用类记得 import**（PERF-04 漏 `java.util.Iterator`）。checkstyle 单独 `mvn checkstyle:check -pl fe-connector/fe-connector-iceberg`。 -6. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。 -7. **并发探测**：本目录 `find fe -newermt '-120 sec'` + `pgrep -a -f maven`。`/mnt/disk1/yy/git/doris` 是另一 worktree，不冲突。 -8. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 +1. **iceberg 侧测试**：`mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`（绝对 `-f /fe/pom.xml`）。原因见旧注：`${revision}` 未 flatten → 必须 `-am`；`-am test` 不产 shade jar 故用 **`install`**；`-Dtest=<类列表>`（`ls src/test/.../*Test.java` 拼逗号，本轮 56 类，存 `scratchpad/iceberg-tests.txt`）让上游快跳。 +2. **⚠ 改到 fe-core 的项须单独验 fe-core**（PERF-06 实证）：**iceberg 连接器不依赖 fe-core**（SPI 解耦），`-pl iceberg -am` 反应堆里**没有 fe-core**——fe-core 的改动 + 其单测须**另跑** `mvn test -pl fe-core -am -Dtest= -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -f /fe/pom.xml`。PERF-07 若动 fe-core sink 流程，两段都要验。 +3. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 静默跳过）。 +4. **别用 `mvn -q` 跑测试**：成功时抑制 `BUILD SUCCESS`/`Tests run` INFO 行。用 surefire XML 或 grep `Tests run:` 聚合验证。 +5. **别 `nohup ... &` 套 `run_in_background`**——maven 变孤儿、"exit 0" 假信号。直接 `mvn ... >> log 2>&1`（run_in_background）；**通知里的 exit code 是 echo 的**，要 grep 日志 `BUILD SUCCESS`。 +6. **checkstyle 主源 ≤120**（test 源 suppress）；import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行；新用类记得 import（本轮四文件各加 `java.util.function.UnaryOperator`）。 +7. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。 +8. **并发探测**：本目录 `find fe -newermt '-120 sec'` + `pgrep -a -f maven`。`/mnt/disk1/yy/git/doris` 是另一 worktree，不冲突。 +9. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 --- # 🗂 开放问题 -- **PERF-06 路由（连接器 hoist vs fe-core 框架 memo）**：`DefaultConnectorContext` 在 fe-core。须先核 `ConnectorContext` API 能否连接器侧 hoist token→map 派生；若须改 fe-core 才能 hoist，则 fe-core 框架层 token-memo（惠及所有 vended 连接器、非 source-specific，PERF-09 先例大概率合规，但须双不变证明）可能更干净。**动码前把方案上交用户定**。 +- **PERF-07 路由（纯连接器 statement-memo vs 动 fe-core sink 绑定流程）**：写路径 resolve 横跨 Nereids 物理算子（fe-core）与 `IcebergWritePlanProvider`（连接器）。先核能否在 `IcebergWritePlanProvider` 内按 queryId/statementId 收敛（连接器侧）；若"resolve 一次传递"须跨 fe-core sink 层，则**动码前把方案上交用户定**（对齐 PERF-06 先例）。 +- **refresh 必要性**：`beginWrite` 无条件 refresh 是否可省/可收敛为语句级 pin 一次，须与 PERF-01 读缓存的命中兼得 + 证写语义 parity。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-06-vended-uri-normalizer-hoist-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-06-vended-uri-normalizer-hoist-design.md new file mode 100644 index 00000000000000..ab6dc8dc678667 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-06-vended-uri-normalizer-hoist-design.md @@ -0,0 +1,106 @@ +# FIX-PERF-06 设计 — vended-credentials 每文件重建 StorageProperties 的 scan 级提升 + +> 覆盖审计发现 **C3**（P1）。路线由用户 2026-07-18 拍板：**连接器侧「准备一次、逐文件套用」**（新增一个通用归一化接口），非 fe-core 框架 memo。 +> 行号基线 = HEAD `27903c9e5d1`（复核已重 grep，见下）。 + +--- + +## Problem + +REST catalog + `iceberg.rest.vended-credentials-enabled=true` 时，扫描规划期每处理**一个 data file、每一个 delete file**，都要把整个 scan 内不变的 vended token 重新翻成一整套 `StorageProperties`（含遍历所有 provider + 新建 hadoop `Configuration` + 逐 key `set` + `StorageProperties.createAll` 推导 region/endpoint）。50k 文件 ≈ 数十秒纯 FE CPU，MOR（带 delete）翻倍。 + +## Root Cause（复核确认，HEAD 行号） + +- 贵活 `DefaultConnectorContext.buildVendedStorageMap(token)`（`DefaultConnectorContext.java:225-242`）**只依赖 token，与 `rawUri` 无关**；`rawUri` 仅用于其后廉价的 `LocationPath.of(rawUri, effective)`（`normalizeStorageUri:392-409`）。 +- vended token 一次 scan 只提取一次，且是**同一个 Map 实例**穿过所有归一化调用： + - 同步数据路径 `planScanInternal:647` → `buildRangeForTask:750` → `buildRange:1131` → `normalizeUri(rawDataPath, vendedToken):1191` + `buildDeleteFiles:1215` → `convertDelete:1242` → `normalizeUri(delete.path, vendedToken):1243`。 + - 流式路径 `streamSplits:496` → `IcebergStreamingSplitSource` 字段 `vendedToken:543` → `buildRangeForTask:578`（同上）。 + - COUNT(*) 下推 `planCountPushdown:1084` → `buildRange`（同上）。 + - 位置删除系统表 `:904` → `buildPositionDeleteRange:929` → `normalizeUri(originalPath, vendedToken):935`。 +- 在这些 seam 里 `vendedToken` **只服务于 `normalizeUri`**（props 生成走另一条 `getScanNodeProperties → vendStorageCredentials`，每 scan 一次，非本病灶）。 +- `DefaultConnectorContext` 是 **per-catalog、跨查询（含并发）共享**的长生命周期对象（`PluginDrivenExternalCatalog.java:197`）——这正是本设计**不**在其上做跨查询 memo 的原因（并发冲刷退化 + 凭证跨查询保留贴近红线）。 + +## Design（route A：scan 级 normalizer） + +把「token → effective 存储配置」的贵活从 per-file 循环里提升到 scan 开始一次，封装成一个**只做廉价路径归一化**的 `UnaryOperator`，逐文件复用。 + +### 新增 SPI seam（连接器无关，默认无副作用） + +`fe-connector-spi / ConnectorContext.java` 新增 default 方法： + +```java +default UnaryOperator newStorageUriNormalizer(Map rawVendedCredentials) { + // 默认：无引擎机制（离线测试 / 无 context）→ 每次 apply 折回 normalizeStorageUri(rawUri, token)， + // 行为与逐文件调用完全一致，不做提升。任何未 override 的连接器不受影响。 + return rawUri -> normalizeStorageUri(rawUri, rawVendedCredentials); +} +``` + +`fe-core / DefaultConnectorContext.java` override（**唯一做提升的真实现**）： + +```java +@Override +public UnaryOperator newStorageUriNormalizer(Map rawVendedCredentials) { + // 贵活一次：token → effective map（vended 覆盖 static，与 normalizeStorageUri 同一 precedence）。 + Map vended = buildVendedStorageMap(rawVendedCredentials); + Map effective = + vended != null ? vended : storagePropertiesSupplier.get(); + // 逐文件只做廉价归一化（保留 empty-uri 短路 + fail-loud LocationPath），与 normalizeStorageUri 逐字对齐。 + return rawUri -> Strings.isNullOrEmpty(rawUri) + ? rawUri + : LocationPath.of(rawUri, effective).toStorageLocation().toString(); +} +``` + +`fe-connector-iceberg / TcclPinningConnectorContext.java` 增 override，透传到真 delegate（否则 SPI 默认会用本 wrapper 的 `normalizeStorageUri` 逐文件折回，拿不到 `DefaultConnectorContext` 的提升）： + +```java +@Override +public UnaryOperator newStorageUriNormalizer(Map rawVendedCredentials) { + return delegate.newStorageUriNormalizer(rawVendedCredentials); +} +``` + +> paimon / hive 的 `TcclPinningConnectorContext` **不动**：它们的连接器不调用新方法，继承 SPI 默认（逐文件折回其自身 `normalizeStorageUri`）= 零回归。 + +### 连接器侧线路改造（`IcebergScanPlanProvider.java`） + +- 新增 package-private helper（供三处 scan-start 与单测统一构造）： + ```java + UnaryOperator newUriNormalizer(Map vendedToken) { + return context != null ? context.newStorageUriNormalizer(vendedToken) : UnaryOperator.identity(); + } + ``` +- 把 per-file seam 的参数 `Map vendedToken` **替换为** `UnaryOperator uriNormalizer`： + `buildRangeForTask` / `buildRange` / `buildDeleteFiles` / `convertDelete` / `buildPositionDeleteRange` / `planCountPushdown`，以及 `IcebergStreamingSplitSource` 的字段 `vendedToken`。 +- 三处 scan-start（`planScanInternal:647`、`streamSplits:496`、位置删除 `:904`）在 `extractVendedToken(...)` 之后立刻 `UnaryOperator uriNormalizer = newUriNormalizer(vendedToken);`，往下传 `uriNormalizer`。 +- 终点 `normalizeUri(path, vendedToken)` → `uriNormalizer.apply(path)`；旧私有 helper `normalizeUri`（1308）随之删除（无其它引用）。 + +**为何是替换而非新增参数**：这些 seam 里 `vendedToken` 的唯一用途就是 `normalizeUri`；替换后不变量在类型上显式化（seam 携带的是"已准备好的归一化器"），也不留悬空的 token 参数。 + +## Parity 论证（byte 不变） + +1. 逐文件输出：`normalizeStorageUri(rawUri, token)` 与 `newStorageUriNormalizer(token).apply(rawUri)` 对**同一 (rawUri, token)** 逐字相同——两者都是 `empty→原样` 否则 `LocationPath.of(rawUri, effective).toStorageLocation().toString()`，`effective` precedence 相同（vended 非空→vended，否则 static）。 +2. `buildVendedStorageMap` fail-soft（catch 返回 null，永不抛）；fail-loud 只在 per-uri 的 `LocationPath.of` —— 提升后 fail-loud 时机不变（仍逐文件）。唯一差异：空文件 scan 下也会跑一次 `buildVendedStorageMap`（fail-soft、廉价、罕见），无可观测行为差。 +3. `storagePropertiesSupplier.get()`（catalog 静态 map）scan 内稳定，capture 一次正确（且顺带省掉逐文件重取）。 +4. 离线测试（`RecordingConnectorContext` 未 override 新方法）继承默认逐文件折回 → `normalizeCount` / `normalizedUris` / `lastVendedToken` 全部照旧触发，现存 recording 断言零改动。 + +## Implementation Plan + +1. SPI：`ConnectorContext.java` 加 default `newStorageUriNormalizer` + import `UnaryOperator`。 +2. fe-core：`DefaultConnectorContext.java` override（复用现成 `buildVendedStorageMap` / `storagePropertiesSupplier` / `LocationPath` / `Strings`）+ import `UnaryOperator`。 +3. iceberg：`TcclPinningConnectorContext.java` override 透传 + import。 +4. iceberg：`IcebergScanPlanProvider.java` 加 `newUriNormalizer` helper、三处 scan-start 构造、六个 seam 换参、删 `normalizeUri` helper、import `UnaryOperator`。 +5. 测试见下。 + +## Test Plan（闸门 = parity + 证明减负） + +- **现存全绿**（parity）：iceberg 全模块 UT（含 `convertDelete*` / vended-token / streaming / count-pushdown / position_deletes 那批）。 +- **改签名的单测**（机械对齐）：`convertDelete(delete, Collections.emptyMap())` → `convertDelete(delete, UnaryOperator.identity())`（仅读 bounds/format/fieldId 的 6 处）；两处断言路径归一化的（`convertDeleteNormalizesDeletePathViaContext`、`convertDeleteNormalizesDeletePathViaVendedToken`）→ `convertDelete(delete, provider.newUriNormalizer(token))`，断言（路径归一化 + `lastVendedToken==token`）不变。 +- **减负守门（新增，SPI 结构层）**：`RecordingConnectorContext` 加 `int newNormalizerCount` 并 override `newStorageUriNormalizer`（自增 + 返回逐文件折回 `normalizeStorageUri` 的 normalizer 以保留现存 recording）。新测：一次 `planScan`（多 data + 多 delete 文件）后断言 `newNormalizerCount == 1` 且 `normalizeCount == N_data + N_delete`——直接编码"token→map 派生每 scan 一次、路径归一化 N 次"。 +- **fe-core parity 单测（新增）**：`DefaultConnectorContext` 的 `newStorageUriNormalizer(token).apply(uri)` 对多个 uri 与 `normalizeStorageUri(uri, token)` 逐字相等（vended 覆盖 + static 回退 + bad-path fail-loud + empty-uri 短路）。 + +## Risk + +- **低**。SPI 加一个默认无副作用方法（其它连接器不受影响）；fe-core override 是把现成 `normalizeStorageUri` 的 body 拆成"准备一次 + 逐文件套用"，无新解析、无新缓存、无跨查询状态、无凭证保留（normalizer 是 scan 局部变量，scan 结束即回收）。 +- 铁律核对：非 source-specific（默认惠及所有 vended 连接器）；fe-core 不新增属性解析（`buildVendedStorageMap` 本就在 fe-core）；不塞 Table 缓存进 fe-core。用户已就"动 fe-core 边界"签字。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-06-vended-uri-normalizer-hoist-summary.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-06-vended-uri-normalizer-hoist-summary.md new file mode 100644 index 00000000000000..27f2ebc7bd0f95 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-06-vended-uri-normalizer-hoist-summary.md @@ -0,0 +1,34 @@ +# FIX-PERF-06 小结 — vended-credentials URI 归一化的 scan 级提升 + +> 覆盖审计发现 **C3**（P1）。commit `6294edf2833`。路线由用户 2026-07-18 拍板：连接器侧「准备一次、逐文件套用」。 + +## Problem + +REST catalog + `iceberg.rest.vended-credentials-enabled=true` 下，扫描规划期每处理**一个 data file、每一个 delete file**都把整个 scan 内不变的 vended token 重新翻成一整套 `StorageProperties`（`StorageProperties.createAll` + 新建 hadoop `Configuration` + 逐 key set）。50k 文件 ≈ 数十秒纯 FE CPU，MOR 翻倍。 + +## Root Cause + +贵活 `buildVendedStorageMap(token)` 只依赖 token、与 per-file URI 无关；token 每 scan 只提取一次且同一 Map 实例穿过所有归一化调用。即：一份完全相同的输入被逐文件重复派生 O(N_files + N_deletes) 次。`DefaultConnectorContext` 是 per-catalog、跨查询共享的长生命周期对象，故**不**在其上做跨查询 memo（并发冲刷退化 + 凭证跨查询保留贴近红线）。 + +## Fix（route A） + +- **SPI**（`ConnectorContext`）新增 default `UnaryOperator newStorageUriNormalizer(token)`：默认逐文件折回 `normalizeStorageUri`（无提升、行为不变，其它连接器零影响）。 +- **fe-core**（`DefaultConnectorContext`）override：把 token→effective 存储配置的贵活**惰性做一次并 memo**，逐文件只做廉价 `LocationPath.of(rawUri, effective)`。逐字对齐原逐文件版：空 URI 短路、vended 覆盖 static、坏路径 fail-loud、异常时机（惰性→首个非空 URI 才派生）全部不变。normalizer 是 scan 局部对象、单线程套用、scan 结束即回收——无跨查询状态、无凭证保留。 +- **iceberg**（`TcclPinningConnectorContext`）override 透传到真 delegate（拿到 fe-core 的提升；此路径纯 fe-core、无需 TCCL pin）。 +- **iceberg**（`IcebergScanPlanProvider`）三处 scan-start（同步 data / 流式 / position_deletes）在 extractVendedToken 后构造一次 normalizer；六个 per-file seam（buildRangeForTask/buildRange/buildDeleteFiles/convertDelete/buildPositionDeleteRange/planCountPushdown + streaming source 字段）参数由 token 换成 normalizer；旧 `normalizeUri` helper 删除。 +- 写路径 `getBackendFileType`（每写一次、非 per-file）不在范围。 + +## Tests + +- **iceberg 全模块 974 UT 绿 / 0 fail / 1 skip**（含迁移的 8 处 convertDelete 签名对齐 + 两处路径归一化 parity 断言保留）。 +- **减负守门（新增）**：`planScanDerivesUriNormalizerOncePerScanNotPerFile` —— 一次扫 3 文件断言 `newNormalizerCount==1`（token→config 派生每 scan 一次）且 `normalizeCount==3`（路径归一化 N 次）。`RecordingConnectorContext` 未 override 前继承 SPI 默认逐文件折回，现存 recording 断言零改动。 +- **fe-core parity（新增，`DefaultConnectorContextNormalizeUriTest` 9→13 UT 绿）**：normalizer.apply 逐字等于 `normalizeStorageUri(uri, token)`（vended 覆盖 / static 回退 / 空URI短路 / 坏路径 fail-loud）+ 一个 normalizer 服务多个 URI。 + +## Result + +一次 vended scan 内 `buildVendedStorageMap`/`StorageProperties.createAll` 从 O(N_files+N_deletes) 次降到 1 次；50k 文件的数十秒 FE CPU 空耗消除。行为对 vended 与非 vended 目录均不变。 + +## 可复用产物 + +- **gate 判据延伸**：scan 内复用（token 恒定）非跨查询，不涉凭证泄漏——与 PERF-05 的跨查询凭证 gate 正交（此处安全因作用域是 scan 而非 catalog）。 +- **SPI「准备一次、套用多次」范式**：`newStorageUriNormalizer` 可被其它 vended 连接器（paimon 每文件亦 normalize）按需 opt-in；默认无副作用。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index 8f1f30252fd2de..f6427f96d5f05c 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -85,3 +85,17 @@ - **结果**：全 iceberg 模块 **973 pass / 0 fail / 0 error / 1 skip**，checkstyle 0 违规，0 回归。summary 见 `designs/FIX-PERF-05-*-summary.md`。 - **新判据（可复用）**：**「缓存 gate 是授权决策，不止凭证泄漏决策」**——session=user 的授权发生在 load 调用里，缓存命中会绕过它，故即便缓存值不含凭证也不能对 session=user 共享；与「值含 FileIO/凭证才 gate」正交，二者叠加判定。 - **下一步**：见 HANDOFF —— PERF-06（C3 REST vended-credentials 每 data/delete file 重建 StorageProperties+Configuration）。 + +--- + +## 2026-07-18 — session 4：PERF-06 实现 + 全绿（commit `6294edf2833`） + +- **复核（HEAD grep，行号已漂）**：确认贵活 `buildVendedStorageMap(token)` 纯 token 派生、与 per-file URI 无关（URI 只用于其后廉价 `LocationPath.of`）；token 每 scan 只提取一次且同一 Map 实例穿三路径（同步 data `planScanInternal`、流式 `streamSplits`、position_deletes）所有 `normalizeUri`（数据 1191/delete 1243/位置删除 935）。**关键生命周期事实**：`DefaultConnectorContext` 是 **per-catalog、跨查询（含并发）共享**（`PluginDrivenExternalCatalog:197`，已缓存 catalogFileSystem）——这决定了不能在其上做跨查询 memo。 +- **路线上交用户（AskUserQuestion）**：(A) 连接器侧「准备一次、逐文件套用」新增 SPI normalizer seam vs (B) fe-core 框架内按 token 单条目 memo。讲清 B 在 per-catalog 共享对象上并发冲刷退化 + 凭证跨查询保留贴红线；Trino 架构参考=每 scan 建一次 FileSystem 复用（对齐 A）。**用户选 A**。 +- **设计红队（独立 Explore agent，6 点核查）**：判 sound、**无 hard blocker**。确认①seam 完整（三路径全覆盖、`planSystemTableScan` 非 position_deletes 分支不 normalize）②parity 逐字（`buildVendedStorageMap` 纯函数 fail-soft、`storagePropertiesSupplier` scan 内稳定甚至更一致、空URI短路保序）③token 不变量④`TcclPinning` override 是拿到提升的必要条件且无需 TCCL pin⑤写路径 `getBackendFileType` O(1)/写、leg"在范围外正确⑥单线程套用无并发隐患。**采纳其加固**：`effective` 改**惰性 memo（首个非空 URI 才派生）**而非构造时 eager，消除"零文件/全空 URI 且 storage-map init 抛"的极窄异常时机分歧。另提示测试迁移 + `context==null` 用 `identity()` 兜底（均已计划）。 +- **实现（跨 3 模块一个 `[perf]` commit）**：SPI `ConnectorContext.newStorageUriNormalizer` default（逐文件折回）；`DefaultConnectorContext` override（惰性 memo 匿名 `UnaryOperator`）；iceberg `TcclPinning` 透传；`IcebergScanPlanProvider` 新增 `newUriNormalizer` helper + 三处 scan-start 构造 + 六 seam 参数 token→normalizer + 删旧 `normalizeUri` helper。 +- **测试**：连接器守门 `planScanDerivesUriNormalizerOncePerScanNotPerFile`（3 文件 scan → `newNormalizerCount==1` & `normalizeCount==3`）；`RecordingConnectorContext` +计数器 + override（仍逐 apply 折回 recording，现存断言零改）；迁移 8 处 convertDelete 签名（6 plain→`identity()`，2 归一化→`newUriNormalizer(token)` 保留 recording 断言）；fe-core `DefaultConnectorContextNormalizeUriTest` +4 parity（vended/static/空URI/坏路径 fail-loud + 一 normalizer 多 URI）。 +- **构建坑（实证）**：iceberg 连接器**不依赖 fe-core**（SPI 解耦），故 `-pl iceberg -am` 反应堆里**无 fe-core**——fe-core override + 其测试须**单独** `mvn test -pl fe-core -am` 验（本任务首个动 fe-core 的 perf 项，后续动 fe-core 者都要两段验）。 +- **结果**：iceberg 全模块 **974 pass / 0 fail / 1 skip**（+1 守门）；fe-core `DefaultConnectorContextNormalizeUriTest` **9→13 绿**；两处 BUILD SUCCESS + checkstyle 0 违规，0 回归。summary 见 `designs/FIX-PERF-06-*-summary.md`。 +- **新判据（可复用）**：**「作用域即安全边界」**——同一份"含凭证派生"的缓存，跨查询（catalog 级对象）做=撞凭证/授权 gate；**scan 级**做=天然安全（token 恒定、无跨用户、结束即回收）。PERF-05 的 gate 是"跨查询缓存必须判授权"，PERF-06 是"把作用域收回 scan 就不必判"——两面。 +- **下一步**：见 HANDOFF —— PERF-07（C20 写路径一条 DML 3~5 次 load 同表）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index a769d116a9814a..f832f89caf0095 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -20,7 +20,7 @@ | PERF-03 | P0 | C2 C11 | #64134 复活：`file_format_type` 兜底走整表 planFiles → 跨查询 `(table,snapshotId)` memoize(连接器侧,无 gate);~~从枚举反推~~已否决(getFileFormatType 早于 planScan + 无过滤 vs 带谓词破 parity) | 与 01/02 共享快照 pin | ✅ 完成 | `0b96f2e6c78` | | PERF-04 | P1 | C17 C18 | streaming / COUNT(*) 下推旁路 IcebergManifestCache → 抽惰性 `cacheBackedFileScanTasks`(delete 索引 eager + data manifest 惰性扁平映射)三处复用;缓存与不 OOM 兼得(否决审计"退回物化"因重引 OOM) | — | ✅ 完成 | `2e5f393779c` | | PERF-05 | P1 | C9 | information_schema.tables 每表 loadTable 取 comment → 复核缩小(普通目录 PERF-01 已覆盖);补**无 gate** `IcebergCommentCache` **仅 vended 非 session=user**(红队:session=user 缓存绕过 per-user 授权=泄漏) | — | ✅ 完成 | `aea3ebdd40e` | -| PERF-06 | P1 | C3 | REST vended-cred 每 data/delete file 重建 StorageProperties+Configuration → scan 级 memo | — | ⏳ | | +| PERF-06 | P1 | C3 | REST vended-cred 每 data/delete file 重建 StorageProperties+Configuration → scan 级「准备一次、逐文件套用」normalizer(SPI seam+fe-core override,连接器侧 hoist);~~fe-core 框架 memo~~否决(per-catalog 跨查询共享→并发冲刷退化+凭证保留贴红线) | — | ✅ 完成 | `6294edf2833` | | PERF-07 | P1 | C20 | 一条 DML 3~5 次 load 同表 → 语句级 resolve 一次传递 | — | ⏳ | | | PERF-08 | P2 | C19 C21 | 维护路径逐单位重扫 / 无去重（rewrite_data_files 每 group planFiles；expire_snapshots S×M）→ union 一次注册 / 按 path 去重 | 改动小可先行 | ⏳ | | | PERF-09 | P2 | C5 | **（fe-core 框架层）** streaming pump 逐 split 重建 backend 候选集 + 锁往返 → 微批 | ⚠ 跨连接器，见约束 | ⏳ | | @@ -65,9 +65,11 @@ - **落地**：无 gate `IcebergCommentCache`（键 TableIdentifier、值 comment String），**仅 vended 且非 session=user 时建**。**红队 HIGH**：`tableCache==null` 做 gate 会卷入 session=user，其授权在 per-user loadTable 里、缓存绕过=元数据泄漏 → 收窄为 vended-only。首次 N load / view 未缓存记为诚实局限。 - **收益**：vended 目录重复 information_schema 从次次 N load→命中；普通/session=user 不变。全模块 973 UT 绿。 -### [ ] PERF-06 — C3：vended-credentials 每文件重建 StorageProperties -- **病灶**：`buildRange:1105 normalizeUri` / `convertDelete:1157 → DefaultConnectorContext.normalizeStorageUri:392-409 → buildVendedStorageMap:225-242 → StorageProperties.createAll`（遍历所有 provider + 建 hadoop `Configuration` + 逐 key set），**每 data file 和每 delete file 各一次**，而 vended token 整个 scan 内不变。50k 文件 ≈ 数十秒纯 FE CPU。门槛：REST + `iceberg.rest.vended-credentials-enabled=true`（MOR 加倍）。 -- **修复方向**：token→typed-map 推导按 scan 提升 / 在 `DefaultConnectorContext` 内做单条目 memo（token 恒等键）。 +### [x] PERF-06 — C3：vended-credentials 每文件重建 StorageProperties · ✅ `6294edf2833` +- **病灶**：`buildRange normalizeUri` / `convertDelete` / `buildPositionDeleteRange` → `DefaultConnectorContext.normalizeStorageUri → buildVendedStorageMap → StorageProperties.createAll`（遍历所有 provider + 建 hadoop `Configuration` + 逐 key set），**每 data file 和每 delete file 各一次**，而 vended token 整个 scan 内不变（同一 Map 实例、贵活纯 token 派生与 URI 无关）。50k 文件 ≈ 数十秒纯 FE CPU。门槛：REST + `iceberg.rest.vended-credentials-enabled=true`（MOR 加倍）。 +- **路线决策（用户 2026-07-18 拍板 route A）**：连接器侧「准备一次、逐文件套用」——新增 SPI `ConnectorContext.newStorageUriNormalizer(token)`（默认逐文件折回、零影响其它连接器），`DefaultConnectorContext` override 惰性 memo 一次派生。**否决 fe-core 框架 memo**：`DefaultConnectorContext` per-catalog 跨查询共享，单条目 memo 并发冲刷退化 + 凭证跨查询保留贴红线；scan 级作用域天然规避（对齐 Trino 每 scan 建一次 FileSystem）。 +- **落地**：三处 scan-start（同步/流式/position_deletes）构造一次 normalizer 往下传，六个 per-file seam 参数由 token 换成 normalizer；`TcclPinning` 透传拿到提升；写路径 `getBackendFileType`（O(1)/写）不动。惰性 memo 对齐原逐文件版空URI短路/vended覆盖/坏路径 fail-loud/异常时机。 +- **收益/守门**：一次 vended scan 内 `buildVendedStorageMap`/`createAll` 从 O(N_files+N_deletes)→1。连接器守门测试断言 3 文件 scan `newNormalizerCount==1` 且 `normalizeCount==3`；fe-core parity 测试断言逐字等价。iceberg 974 UT 绿 + fe-core NormalizeUriTest 13 绿。 ### [ ] PERF-07 — C20：写路径 3~5 次 load 同表 - **病灶**：`PhysicalIcebergMergeSink.getRequirePhysicalProperties:161→198`、`PhysicalPlanTranslator.visitPhysicalConnectorTableSink:675,703`、`PluginDrivenTableSink.bindDataSink:175 → IcebergWritePlanProvider.resolveTable:689-702 / beginWrite`（内含 tableExists×2 + 无条件 refresh）。每条 DML +3~5 次串行远程往返。 From ca54331b9ae1e1212417832bbc469e612124a870 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 20:47:20 +0800 Subject: [PATCH 16/47] [perf](catalog) fe-core: add a per-statement ConnectorStatementScope (PERF-07) Introduce ConnectorStatementScope, a neutral SPI memoization arena that lives exactly as long as one SQL statement, so a connector can load a table (and derive per-statement state) once and share that one object across every read + write resolver in the statement. - fe-connector-api: ConnectorStatementScope (interface + NONE no-op) and a default ConnectorSession.getStatementScope() returning NONE, so all existing session implementations are unaffected. - fe-core: ConnectorStatementScopeImpl (ConcurrentHashMap-backed); a lazily built StatementContext field with a synchronized accessor plus a reset (not cleared on close/release, reclaimed with the statement, mirroring snapshots); ConnectorSessionImpl/Builder capture the scope reference at construction (the request thread, where the statement context is reachable) so off-thread scan pumps that reuse the one session still reach it; and ExecuteCommand resets the scope on each prepared-statement execution so a reused context starts fresh. This is the fe-core landing point for Trino's per-transaction metadata-cache model, which Doris cannot host on the ~26-times-rebuilt connector session but can hang on the one per-statement StatementContext. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../doris/connector/api/ConnectorSession.java | 14 +++ .../api/ConnectorStatementScope.java | 54 +++++++++++ .../connector/ConnectorSessionBuilder.java | 42 +++++++- .../doris/connector/ConnectorSessionImpl.java | 15 ++- .../ConnectorStatementScopeImpl.java | 45 +++++++++ .../doris/nereids/StatementContext.java | 33 +++++++ .../trees/plans/commands/ExecuteCommand.java | 4 + .../ConnectorStatementScopeTest.java | 95 +++++++++++++++++++ 8 files changed, 299 insertions(+), 3 deletions(-) create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatementScope.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java index 9d23bde4a32155..320805bacd0faf 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorSession.java @@ -128,4 +128,18 @@ default void setCurrentTransaction(ConnectorTransaction txn) { default long allocateTransactionId() { throw new UnsupportedOperationException("transaction id allocation not supported"); } + + /** + * Returns the per-statement scope for this session — a memoization arena that lives exactly as long + * as the current SQL statement, letting a connector load a table (and derive per-statement state) + * once and share that single object across every read + write resolver in the statement. + * + *

    The default is {@link ConnectorStatementScope#NONE} (no memoization = load every time), so + * every existing implementation is unaffected. The engine session implementation overrides it with a + * scope hung on the statement, captured at session construction (the request thread, where the + * statement context is reachable) so off-thread scan pumps that reuse the session still reach it.

    + */ + default ConnectorStatementScope getStatementScope() { + return ConnectorStatementScope.NONE; + } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatementScope.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatementScope.java new file mode 100644 index 00000000000000..d12d2b45c15485 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatementScope.java @@ -0,0 +1,54 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api; + +import java.util.function.Supplier; + +/** + * A per-statement memoization arena, living exactly as long as one SQL statement (read + write). + * + *

    It lets a connector load a table (and derive per-statement state) once and share that single + * object across every resolver in the statement — read metadata, scan planning, write shaping, + * begin-write. Reached from a connector via {@link ConnectorSession#getStatementScope()}.

    + * + *

    Neutral SPI: the engine owns the physical home (the scope is hung on the engine's per-statement + * context) and the connector stores its own connector-typed values under string keys as opaque + * {@code Object}s — fe-core never sees a connector type. Values are session-bound and reclaimed with + * the statement, never promoted into any cross-session / cross-identity structure.

    + * + *

    {@link #NONE} is the off-context default (no live statement: offline planning, tests, and any + * {@link ConnectorSession} that does not override {@link ConnectorSession#getStatementScope()}). It + * never memoizes, so every call runs the loader — byte-identical to loading every time.

    + */ +public interface ConnectorStatementScope { + + /** + * Returns the value cached under {@code key}, computing it with {@code loader} on first access and + * caching it for the rest of the statement. Within one statement the same key returns the same + * instance to every caller; under {@link #NONE} the loader runs on every call. + */ + T computeIfAbsent(String key, Supplier loader); + + /** The no-op scope: never caches; each call invokes the loader (offline / no-context / tests). */ + ConnectorStatementScope NONE = new ConnectorStatementScope() { + @Override + public T computeIfAbsent(String key, Supplier loader) { + return loader.get(); + } + }; +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionBuilder.java index 717c505d10b52e..f26cbfe22dbf82 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionBuilder.java @@ -21,8 +21,10 @@ import org.apache.doris.common.util.DebugUtil; import org.apache.doris.connector.api.ConnectorDelegatedCredential; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.datasource.DelegatedCredential; import org.apache.doris.datasource.SessionContext; +import org.apache.doris.nereids.StatementContext; import org.apache.doris.qe.ConnectContext; import org.apache.doris.qe.GlobalVariable; import org.apache.doris.qe.VariableMgr; @@ -56,6 +58,9 @@ public final class ConnectorSessionBuilder { // over the ConnectContext extraction. private String sessionId; private ConnectorDelegatedCredential delegatedCredential; + // Explicit per-statement scope override for tests without a live ConnectContext; when set it wins over + // the ConnectContext capture in build(). + private ConnectorStatementScope statementScope; private ConnectorSessionBuilder() {} @@ -139,6 +144,16 @@ public ConnectorSessionBuilder withDelegatedCredential(ConnectorDelegatedCredent return this; } + /** + * Sets the per-statement scope explicitly (for callers without a live {@link ConnectContext}, e.g. tests + * that want to inject a memoizing scope). When unset, {@link #build()} captures it from the originating + * {@link ConnectContext}'s statement context, or falls back to {@link ConnectorStatementScope#NONE}. + */ + public ConnectorSessionBuilder withStatementScope(ConnectorStatementScope statementScope) { + this.statementScope = statementScope; + return this; + } + /** Builds an immutable {@link ConnectorSession} instance. */ public ConnectorSession build() { String sid = null; @@ -159,7 +174,32 @@ public ConnectorSession build() { } } return new ConnectorSessionImpl(queryId, user, timeZone, locale, - catalogId, catalogName, catalogProperties, sessionProperties, sid, cred); + catalogId, catalogName, catalogProperties, sessionProperties, sid, cred, + captureStatementScope()); + } + + /** + * Captures the per-statement scope at build time (the request thread). An explicit test override wins; + * otherwise read it off the originating {@link ConnectContext}'s statement context (preferring the + * retained context from {@link #from} over the thread-local, so a session built off the request thread + * still captures correctly). Two-level null (no context / no statement context) yields + * {@link ConnectorStatementScope#NONE} -- mirrors {@code MvccUtil.getSnapshotFromContext}. Capturing the + * reference here (not reading it live) lets off-thread scan pumps that reuse this one session reach the + * same scope even though they have no ConnectContext thread-local. + */ + private ConnectorStatementScope captureStatementScope() { + if (statementScope != null) { + return statementScope; + } + ConnectContext ctx = connectContext != null ? connectContext : ConnectContext.get(); + if (ctx == null) { + return ConnectorStatementScope.NONE; + } + StatementContext sc = ctx.getStatementContext(); + if (sc == null) { + return ConnectorStatementScope.NONE; + } + return sc.getOrCreateConnectorStatementScope(); } /** diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionImpl.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionImpl.java index 722831d3fcff87..a273b9a9d94b2a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionImpl.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorSessionImpl.java @@ -20,6 +20,7 @@ import org.apache.doris.catalog.Env; import org.apache.doris.connector.api.ConnectorDelegatedCredential; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.handle.ConnectorTransaction; import java.util.Collections; @@ -48,6 +49,10 @@ public class ConnectorSessionImpl implements ConnectorSession { // ConnectorDelegatedCredential); getSessionId() falls back to the queryId when no session id was captured. private final String sessionId; private final ConnectorDelegatedCredential delegatedCredential; + // The per-statement scope, captured at construction (the request thread, where the statement context is + // reachable) so off-thread scan pumps that reuse this one session still reach it. NONE when there is no + // live statement context (offline planning, tests) -- then getStatementScope() memoizes nothing. + private final ConnectorStatementScope statementScope; // Otherwise-immutable session; this is bound once by the insert executor at write time // for connectors using the SPI transaction model (e.g. maxcompute), and read back by the // connector's planWrite via getCurrentTransaction(). volatile for cross-thread visibility. @@ -57,13 +62,13 @@ public class ConnectorSessionImpl implements ConnectorSession { long catalogId, String catalogName, Map catalogProperties, Map sessionProperties) { this(queryId, user, timeZone, locale, catalogId, catalogName, catalogProperties, sessionProperties, - null, null); + null, null, ConnectorStatementScope.NONE); } ConnectorSessionImpl(String queryId, String user, String timeZone, String locale, long catalogId, String catalogName, Map catalogProperties, Map sessionProperties, String sessionId, - ConnectorDelegatedCredential delegatedCredential) { + ConnectorDelegatedCredential delegatedCredential, ConnectorStatementScope statementScope) { this.queryId = queryId != null ? queryId : ""; this.user = user != null ? user : ""; this.timeZone = timeZone != null ? timeZone : "UTC"; @@ -76,6 +81,7 @@ public class ConnectorSessionImpl implements ConnectorSession { ? Collections.unmodifiableMap(sessionProperties) : Collections.emptyMap(); this.sessionId = sessionId; this.delegatedCredential = delegatedCredential; + this.statementScope = statementScope != null ? statementScope : ConnectorStatementScope.NONE; } @Override @@ -174,6 +180,11 @@ public Optional getCurrentTransaction() { return Optional.ofNullable(currentTransaction); } + @Override + public ConnectorStatementScope getStatementScope() { + return statementScope; + } + @Override public String toString() { return "ConnectorSession{queryId='" + queryId diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java new file mode 100644 index 00000000000000..6d40721b0edaba --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java @@ -0,0 +1,45 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector; + +import org.apache.doris.connector.api.ConnectorStatementScope; + +import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Supplier; + +/** + * Statement-scoped memoization arena backing {@link ConnectorStatementScope}, hung on the per-statement + * {@link org.apache.doris.nereids.StatementContext}. + * + *

    Thread-safe by a backing {@link ConcurrentHashMap}: a scan's off-thread pumps (streaming / + * partition-batch) reuse the single {@link org.apache.doris.connector.api.ConnectorSession} built on the + * request thread and so reach this same scope concurrently. {@code computeIfAbsent} gives every caller of + * a key the same instance — required for the shared table object and for the delete supply map that scan + * and write both mutate. The loaders used by connectors do not re-enter this scope, so the map's + * single-key atomicity is safe here.

    + */ +public class ConnectorStatementScopeImpl implements ConnectorStatementScope { + + private final ConcurrentHashMap cache = new ConcurrentHashMap<>(); + + @Override + @SuppressWarnings("unchecked") + public T computeIfAbsent(String key, Supplier loader) { + return (T) cache.computeIfAbsent(key, k -> loader.get()); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java index d3027ed8da3494..9f1e5a47d69c24 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java @@ -30,6 +30,8 @@ import org.apache.doris.common.Id; import org.apache.doris.common.IdGenerator; import org.apache.doris.common.Pair; +import org.apache.doris.connector.ConnectorStatementScopeImpl; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.handle.ConnectorTransaction; import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.mvcc.MvccSnapshot; @@ -198,6 +200,14 @@ public enum TableFrom { private final Set keySlots = Sets.newHashSet(); private BitSet disableRules; + // A per-statement memoization arena for connectors: e.g. Iceberg loads a table once and shares that + // single object across read + write resolvers within the statement. Lazily built (see + // getOrCreateConnectorStatementScope), values stored opaquely by the connector. Like snapshots, it is + // NOT cleared on close()/releasePlannerResources -- it survives to statement GC. A reused + // prepared-statement context drops it per execution via resetConnectorStatementScope() (see + // ExecuteCommand) so one execution's cached tables never leak into the next. + private ConnectorStatementScope connectorStatementScope; + // table locks private final Stack plannerResources = new Stack<>(); @@ -637,6 +647,29 @@ public synchronized BitSet getOrCacheDisableRules(SessionVariable sessionVariabl return this.disableRules; } + /** + * Returns this statement's connector scope, lazily creating it on first use (mirrors + * {@link #getOrCacheDisableRules}). A connector reaches it through + * {@link org.apache.doris.connector.api.ConnectorSession#getStatementScope()} to load a table once and + * share it across the statement's read + write resolvers. + */ + public synchronized ConnectorStatementScope getOrCreateConnectorStatementScope() { + if (this.connectorStatementScope == null) { + this.connectorStatementScope = new ConnectorStatementScopeImpl(); + } + return this.connectorStatementScope; + } + + /** + * Drops the connector scope so the next statement execution starts fresh. Prepared-statement EXECUTE + * reuses one StatementContext across executions (see + * {@link org.apache.doris.nereids.trees.plans.commands.ExecuteCommand}); this is called there, beside + * the other per-execution resets, so a prior execution's cached tables/state never leak into the next. + */ + public synchronized void resetConnectorStatementScope() { + this.connectorStatementScope = null; + } + /** * Some value of the cacheKey may change, invalid cache when value change */ diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommand.java index df50bbf6266ecd..45b5c7d70410ee 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommand.java @@ -89,6 +89,10 @@ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception { StatementContext statementContext = preparedStmtCtx.getStatementContext(); statementContext.setPrepareStage(false); statementContext.setIsInsert(false); + // A prepared EXECUTE reuses this one StatementContext across executions; drop the connector + // per-statement scope so a prior execution's cached tables/state never leak into this one (the + // scope key's queryId is a second line of defense). See StatementContext#resetConnectorStatementScope. + statementContext.resetConnectorStatementScope(); LogicalPlan logicalPlan = prepareCommand.getLogicalPlan(); if (logicalPlan instanceof LogicalSqlCache) { throw new AnalysisException("Unsupported sql cache for server prepared statement"); diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java new file mode 100644 index 00000000000000..a4a6e3726c9e73 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java @@ -0,0 +1,95 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector; + +import org.apache.doris.connector.api.ConnectorStatementScope; +import org.apache.doris.nereids.StatementContext; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Tests for the per-statement {@link ConnectorStatementScope}: the {@link ConnectorStatementScope#NONE} no-op, + * the memoizing {@link ConnectorStatementScopeImpl}, and the {@link StatementContext} hosting + per-execution + * reset a reused prepared statement relies on. + */ +public class ConnectorStatementScopeTest { + + @Test + public void noneNeverMemoizes() { + // NONE = the off-context default (offline / no live statement): the loader runs every call, so a + // connector under NONE behaves byte-identically to loading every time. MUTATION: NONE memoizing -> the + // second call reuses the first value -> red. + AtomicInteger loads = new AtomicInteger(); + ConnectorStatementScope none = ConnectorStatementScope.NONE; + Object first = none.computeIfAbsent("k", () -> { + loads.incrementAndGet(); + return new Object(); + }); + Object second = none.computeIfAbsent("k", () -> { + loads.incrementAndGet(); + return new Object(); + }); + Assertions.assertEquals(2, loads.get(), "NONE must run the loader every time (no memo)"); + Assertions.assertNotSame(first, second, "NONE returns a fresh value each call"); + } + + @Test + public void implMemoizesPerKeyAndIsolatesKeys() { + // The real scope memoizes per key: the same key returns the same instance to every caller (this is what + // makes a statement's read/scan/write resolvers share ONE loaded table), and the loader runs once. + // MUTATION: not memoizing -> two instances / two loads -> red. + ConnectorStatementScope scope = new ConnectorStatementScopeImpl(); + AtomicInteger loads = new AtomicInteger(); + Object firstA = scope.computeIfAbsent("A", () -> { + loads.incrementAndGet(); + return new Object(); + }); + Object secondA = scope.computeIfAbsent("A", () -> { + loads.incrementAndGet(); + return new Object(); + }); + Assertions.assertSame(firstA, secondA, "the same key returns the same instance (read/write share one load)"); + Assertions.assertEquals(1, loads.get(), "the loader runs once per key"); + + Object b = scope.computeIfAbsent("B", Object::new); + Assertions.assertNotSame(firstA, b, "different keys are isolated (cross-catalog / cross-table)"); + } + + @Test + public void statementContextScopeIsStableThenResetsForReusedPreparedContext() { + // StatementContext lazily builds one scope and reuses it across the statement (getOrCreate...). A prepared + // EXECUTE reuses one StatementContext across executions; resetConnectorStatementScope() (called by + // ExecuteCommand each execution) drops it so a prior execution's memoized state never leaks into the next. + // MUTATION: reset not clearing the field -> s2 == s1 and the stale value leaks -> red. + StatementContext ctx = new StatementContext(); + ConnectorStatementScope s1 = ctx.getOrCreateConnectorStatementScope(); + Assertions.assertSame(s1, ctx.getOrCreateConnectorStatementScope(), + "the scope is lazily created once and reused across a statement"); + + Object memoized = s1.computeIfAbsent("k", Object::new); + ctx.resetConnectorStatementScope(); + + ConnectorStatementScope s2 = ctx.getOrCreateConnectorStatementScope(); + Assertions.assertNotSame(s1, s2, "reset drops the scope so a reused prepared context starts fresh"); + Assertions.assertNotSame(memoized, s2.computeIfAbsent("k", Object::new), + "a prior execution's memoized value must not leak into the next execution"); + } +} From 33a265c7aa5d03f0e2d925a6dd1385250b4904f2 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 20:47:55 +0800 Subject: [PATCH 17/47] [perf](catalog) fe-connector-iceberg: load each table once per statement, drop the fat handle and the delete stash (PERF-07) Route the read metadata path, scan planning, write shaping and beginWrite through the per-statement ConnectorStatementScope so one statement loads each iceberg table once and every resolver shares that one RAW object; the snapshot pin and the Kerberos doAs FileIO wrap are applied per consumer, never frozen into the shared object. - New IcebergStatementScope keys the shared table by catalogId:db:table:queryId and the rewritable-delete supply by catalogId:queryId; a null/NONE session loads every time (byte-identical to the pre-scope offline behavior). - Remove the fat-handle memo (IcebergTableHandle.resolvedTable + accessors + the three with* carries); the handle is now pure coordinates. - beginWrite resolves the shared table instead of loading fresh (a row-level DML reuses the object the scan already loaded); openTransaction keeps newTransaction()'s refresh, giving the commit a fresh OCC base. - Move the merge-on-read rewritable-delete supply out of the per-catalog IcebergRewritableDeleteStash singleton (deleted) into the scope: the scan seam accumulates, the write seam drains, both by the same key. A format-version>=3 DELETE/UPDATE/MERGE under a NONE scope now fails loud instead of silently writing a deletion vector that would drop old deletes and resurrect rows. Read, scan and write of one statement resolve ops through the same per-session newCatalogBackedOps(session), so sharing one table stays within one user's identity (session=user / REST-vended safe). There is no write-side latency goal; the value is architectural coherence and removing the singleton + class. iceberg 968 pass / 1 skip; fe-core scope + session tests green. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../connector/iceberg/IcebergConnector.java | 15 +- .../iceberg/IcebergConnectorMetadata.java | 67 +++--- .../iceberg/IcebergConnectorTransaction.java | 6 +- .../iceberg/IcebergManifestCache.java | 2 +- .../iceberg/IcebergRewritableDeleteStash.java | 140 ------------ .../iceberg/IcebergScanPlanProvider.java | 107 ++++----- .../iceberg/IcebergStatementScope.java | 84 ++++++++ .../connector/iceberg/IcebergTableHandle.java | 41 +--- .../iceberg/IcebergWritePlanProvider.java | 71 +++--- .../IcebergProviderSessionRoutingTest.java | 2 +- .../IcebergRewritableDeleteStashTest.java | 204 ------------------ .../iceberg/IcebergScanPlanProviderTest.java | 98 ++++++--- .../iceberg/IcebergStatementScopeTest.java | 179 +++++++++++++++ .../iceberg/IcebergTableHandleTest.java | 59 ----- .../iceberg/IcebergWritePlanProviderTest.java | 124 ++++++----- .../connector/iceberg/TestStatementScope.java | 41 ++++ 16 files changed, 576 insertions(+), 664 deletions(-) delete mode 100644 fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java create mode 100644 fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java delete mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStashTest.java create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergStatementScopeTest.java create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TestStatementScope.java diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java index 172bfba369024d..bbc2ca71174f2c 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java @@ -161,8 +161,9 @@ public class IcebergConnector implements Connector { private final IcebergLatestSnapshotCache latestSnapshotCache; // PERF-01: cross-query cache of the RAW iceberg Table (restores the legacy IcebergExternalMetaCache table // cache that the SPI cutover dropped). null when the catalog's credentials are query-dependent - // (iceberg.rest.session=user / REST vended-credentials) — see the constructor. The query-scoped fat handle - // (IcebergTableHandle.resolvedTable) is always on and independent of this field. + // (iceberg.rest.session=user / REST vended-credentials) — see the constructor. The per-statement scope + // (ConnectorStatementScope) shares one loaded table across a statement's read/scan/write regardless of this + // field, and is what a credential-gated catalog (this field null) relies on within a statement. private final IcebergTableCache tableCache; // PERF-02: cross-query partition-view cache (the raw PARTITIONS-scan result, keyed by (table, snapshotId)). // Built unconditionally — the cached value is pure metadata with no FileIO/credential, so no credential gate @@ -178,11 +179,6 @@ public class IcebergConnector implements Connector { // authorization a shared cache would bypass. null for every other flavor. private final IcebergCommentCache commentCache; private final IcebergManifestCache manifestCache = new IcebergManifestCache(); - // commit-bridge supply (S4 part 2): per-catalog stash carrying a row-level DML's non-equality delete supply - // across the scan->write seam — the scan provider fills it (keyed by queryId), the write provider drains it - // into rewritable_delete_file_sets. Like the caches above, a REFRESH CATALOG rebuilds the connector and thus - // drops it. Inert pre-cutover (iceberg scans/writes do not route through the providers until P6.6). - private final IcebergRewritableDeleteStash rewritableDeleteStash = new IcebergRewritableDeleteStash(); // Lazily-built plugin-side Kerberos authenticator (single-owner auth; see TcclPinningConnectorContext). // null for a non-Kerberos catalog. Its doAs acts on the PLUGIN's UserGroupInformation copy — the one the @@ -679,7 +675,7 @@ public ConnectorScanPlanProvider getScanPlanProvider() { // threaded for parity with the legacy single per-catalog IcebergMetadataOps. return new IcebergScanPlanProvider(properties, this::newCatalogBackedOps, context, manifestCache, - rewritableDeleteStash, tableCache, formatCache); + tableCache, formatCache); } @Override @@ -689,8 +685,7 @@ public ConnectorWritePlanProvider getWritePlanProvider() { // IcebergConnectorTransaction. It resolves the target via catalogOps.loadTable, so it shares the // fully-threaded ops (newCatalogBackedOps) — external_catalog.name must apply to INSERT/DELETE/MERGE. return new IcebergWritePlanProvider(properties, - this::newCatalogBackedOps, context, - rewritableDeleteStash); + this::newCatalogBackedOps, context); } @Override diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index 3aadcd2ff4e7b1..f999b8adfdd055 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -353,13 +353,13 @@ public String getTableComment(ConnectorSession session, String dbName, String ta // handle) is not cached and propagates to the caller's catch (still ""), so behavior is unchanged. if (commentCache != null) { return commentCache.getOrLoad(TableIdentifier.of(dbName, tableName), - () -> loadTableComment(dbName, tableName)); + () -> loadTableComment(session, dbName, tableName)); } - return loadTableComment(dbName, tableName); + return loadTableComment(session, dbName, tableName); } - private String loadTableComment(String dbName, String tableName) { - Table table = loadTable(new IcebergTableHandle(dbName, tableName)); + private String loadTableComment(ConnectorSession session, String dbName, String tableName) { + Table table = loadTable(session, new IcebergTableHandle(dbName, tableName)); return table.properties().getOrDefault(TABLE_COMMENT_PROP, ""); } @@ -394,7 +394,7 @@ public ConnectorTableSchema getTableSchema( } // Mirror legacy IcebergMetadataOps.loadTable: wrap the remote load in the auth context. The schema // + table-property assembly is pure (operates on the already-loaded Table). - Table table = loadTable(iceHandle); + Table table = loadTable(session, iceHandle); return buildTableSchema(iceHandle.getTableName(), table, table.schema()); } @@ -419,7 +419,7 @@ public ConnectorTableSchema getTableSchema( if (snapshot == null || snapshot.getSchemaId() < 0) { return getTableSchema(session, handle); } - Table table = loadTable(iceHandle); + Table table = loadTable(session, iceHandle); Schema schema; if (table.currentSnapshot() == null) { // Empty table: legacy getSchema falls back to the latest schema (NEWEST_SCHEMA_ID path). @@ -591,35 +591,30 @@ static String buildShowSortClause(Table table) { * remote), so the many reads sharing one handle in a planning/analysis pass collapse onto a single remote * {@code loadTable} (PERF-01); a fat-handle hit returns without any remote call. */ - private Table loadTable(IcebergTableHandle handle) { + private Table loadTable(ConnectorSession session, IcebergTableHandle handle) { try { - return context.executeAuthenticated(() -> resolveTableForRead(handle)); + return context.executeAuthenticated(() -> resolveTableForRead(session, handle)); } catch (Exception e) { throw new RuntimeException("Failed to load table, error message is:" + e.getMessage(), e); } } /** - * Resolves the RAW iceberg {@link Table} for {@code handle} with PERF-01's two-layer memo, WITHOUT opening - * an auth scope or wrapping exceptions — callers own both. The query-scoped fat handle comes first (same - * handle instance -> same table); then the cross-query {@link IcebergTableCache} when enabled (else a - * direct remote load); finally the result is memoized back onto the handle. The remote loader's exception - * propagates verbatim (the cache re-throws it unwrapped), so a caller's own {@code NoSuchTableException} - * degradation (the partition-view readers) still fires. Callers needing the auth scope wrap the call in - * {@code executeAuthenticated} (see {@link #loadTable}). NOT used by the sys-table path - * ({@link #loadSysTable}) or the DDL/write path (both take a fresh remote base by design). + * Resolves the RAW iceberg {@link Table} for {@code handle}, WITHOUT opening an auth scope or wrapping + * exceptions — callers own both. The per-statement scope ({@link IcebergStatementScope#sharedTable}) comes + * first, so the statement's read metadata, scan planning and write all resolve the SAME one loaded object; + * on a scope miss the loader consults the cross-query {@link IcebergTableCache} when enabled (else a direct + * remote load). The remote loader's exception propagates verbatim (the cache re-throws it unwrapped), so a + * caller's own {@code NoSuchTableException} degradation (the partition-view readers) still fires. Callers + * needing the auth scope wrap the call in {@code executeAuthenticated} (see {@link #loadTable}). NOT used by + * the sys-table path ({@link #loadSysTable}), which takes a fresh remote base by design. */ - private Table resolveTableForRead(IcebergTableHandle handle) { - Table cached = handle.getResolvedTable(); - if (cached != null) { - return cached; - } - Table table = tableCache != null - ? tableCache.getOrLoad(TableIdentifier.of(handle.getDbName(), handle.getTableName()), - () -> catalogOps.loadTable(handle.getDbName(), handle.getTableName())) - : catalogOps.loadTable(handle.getDbName(), handle.getTableName()); - handle.setResolvedTable(table); - return table; + private Table resolveTableForRead(ConnectorSession session, IcebergTableHandle handle) { + return IcebergStatementScope.sharedTable(session, handle.getDbName(), handle.getTableName(), + () -> tableCache != null + ? tableCache.getOrLoad(TableIdentifier.of(handle.getDbName(), handle.getTableName()), + () -> catalogOps.loadTable(handle.getDbName(), handle.getTableName())) + : catalogOps.loadTable(handle.getDbName(), handle.getTableName())); } /** @@ -660,7 +655,7 @@ public Map getColumnHandles( // Mirror getTableSchema: wrap the remote load in the auth context. A sys handle resolves the // metadata-table columns (t$snapshots -> committed_at/...) so the generic scan node can look up // its pruned sys-table slots by name; a data handle resolves the base table's columns. - Table table = iceHandle.isSystemTable() ? loadSysTable(iceHandle) : loadTable(iceHandle); + Table table = iceHandle.isSystemTable() ? loadSysTable(iceHandle) : loadTable(session, iceHandle); List fields = table.schema().columns(); Map handles = new LinkedHashMap<>(fields.size()); for (Types.NestedField field : fields) { @@ -697,7 +692,7 @@ public Optional getTableStatistics( } long rowCount; try { - rowCount = computeRowCount(loadTable(iceHandle)); + rowCount = computeRowCount(loadTable(session, iceHandle)); } catch (Exception e) { LOG.warn("Failed to compute Iceberg row count for {}.{}", iceHandle.getDbName(), iceHandle.getTableName(), e); @@ -1448,7 +1443,7 @@ public void validateRowLevelDmlMode(ConnectorSession session, ConnectorTableHand default: return; } - Table table = loadTable((IcebergTableHandle) handle); + Table table = loadTable(session, (IcebergTableHandle) handle); String mode = table.properties().getOrDefault(modeProperty, defaultMode); if (RowLevelOperationMode.COPY_ON_WRITE.modeName().equalsIgnoreCase(mode)) { throw new DorisConnectorException(String.format( @@ -1473,7 +1468,7 @@ public void validateStaticPartitionColumns(ConnectorSession session, ConnectorTa return; } IcebergTableHandle iceHandle = (IcebergTableHandle) handle; - Table table = loadTable(iceHandle); + Table table = loadTable(session, iceHandle); PartitionSpec spec = table.spec(); String tableName = iceHandle.getTableName(); if (!spec.isPartitioned()) { @@ -1526,7 +1521,7 @@ public Optional getMvccPartitionView( IcebergTableHandle iceHandle = (IcebergTableHandle) handle; try { return context.executeAuthenticated(() -> { - Table table = resolveTableForRead(iceHandle); + Table table = resolveTableForRead(session, iceHandle); return Optional.of(IcebergPartitionUtils.buildMvccPartitionView(table, iceHandle.getSnapshotId(), TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache)); }); @@ -1550,7 +1545,7 @@ public List listPartitionNames(ConnectorSession session, ConnectorTableH return context.executeAuthenticated(() -> { Table table; try { - table = resolveTableForRead(iceHandle); + table = resolveTableForRead(session, iceHandle); } catch (NoSuchTableException e) { LOG.warn("Iceberg table not found while listing partitions: {}.{}", iceHandle.getDbName(), iceHandle.getTableName(), e); @@ -1582,7 +1577,7 @@ public List listPartitions(ConnectorSession session, return context.executeAuthenticated(() -> { Table table; try { - table = resolveTableForRead(iceHandle); + table = resolveTableForRead(session, iceHandle); } catch (NoSuchTableException e) { LOG.warn("Iceberg table not found while listing partitions: {}.{}", iceHandle.getDbName(), iceHandle.getTableName(), e); @@ -1617,7 +1612,7 @@ public Optional beginQuerySnapshot( IcebergTableHandle iceHandle = (IcebergTableHandle) handle; TableIdentifier id = TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()); IcebergLatestSnapshotCache.CachedSnapshot pin = latestSnapshotCache.getOrLoad(id, () -> { - Table table = loadTable(iceHandle); + Table table = loadTable(session, iceHandle); Snapshot current = table.currentSnapshot(); return new IcebergLatestSnapshotCache.CachedSnapshot( current == null ? -1L : current.snapshotId(), table.schema().schemaId()); @@ -1650,7 +1645,7 @@ public Optional beginQuerySnapshot( @Override public Optional resolveTimeTravel( ConnectorSession session, ConnectorTableHandle handle, ConnectorTimeTravelSpec spec) { - Table table = loadTable((IcebergTableHandle) handle); + Table table = loadTable(session, (IcebergTableHandle) handle); switch (spec.getKind()) { case SNAPSHOT_ID: { long id = Long.parseLong(spec.getStringValue()); diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java index dc6342b0fef3c3..b850f84c63fdc4 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorTransaction.java @@ -205,7 +205,11 @@ public void beginWrite(ConnectorSession session, String db, String tableName, Ic this.zone = IcebergTimeUtils.resolveSessionZone(session); try { context.executeAuthenticated(() -> { - Table loaded = catalogOps.loadTable(db, tableName); + // PERF-07: resolve the table through the per-statement scope so a row-level DML reuses the SAME + // one object the scan already loaded (a write-only INSERT loads once here). openTransaction still + // issues newTransaction()'s refresh, giving the commit a fresh OCC base off that shared object. + Table loaded = IcebergStatementScope.sharedTable(session, db, tableName, + () -> catalogOps.loadTable(db, tableName)); this.table = loaded; applyBeginGuards(ctx, tableName); this.transaction = openTransaction(loaded); diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java index 689f1e7722dad4..f3c46732fb2fc1 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergManifestCache.java @@ -73,7 +73,7 @@ final class IcebergManifestCache { // (ConnectorSession.getQueryId()), so VERBOSE EXPLAIN can report THIS scan's hits/misses/failures (the // "manifest cache:" line). The provider that PLANS the scan and the (transient, fresh-per-call) provider that // renders EXPLAIN are different instances, so per-scan state cannot live on the provider; the long-lived - // per-catalog cache is their only shared survivor — same rationale as IcebergRewritableDeleteStash. + // per-catalog cache is their only shared survivor (the queryId keys this per-scan tally onto it). // {@link #takeStats} is the primary eviction (EXPLAIN drains its entry); a non-EXPLAIN query records but // never drains, so its leaked entry is aged out by the lazy TTL sweep in {@link #touch}. private final Map statsByQuery = new ConcurrentHashMap<>(); diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java deleted file mode 100644 index 3f153371b8bb95..00000000000000 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStash.java +++ /dev/null @@ -1,140 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.connector.iceberg; - -import org.apache.doris.thrift.TIcebergDeleteFileDesc; - -import java.util.List; -import java.util.Map; -import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.TimeUnit; -import java.util.function.LongSupplier; - -/** - * Per-catalog stash that carries the merge-on-read "rewritable delete" supply across the scan→write seam of a - * single row-level DML (DELETE / UPDATE / MERGE) on a format-version≥3 iceberg table. - * - *

    Why a singleton stash. When a DML rewrites the deletes of a data file, the BE writes a NEW deletion - * vector that must OR-merge that file's still-live old deletes (old DVs + old position deletes) — otherwise the - * previously-deleted rows resurrect. The BE does not read iceberg metadata; the FE must hand it, per touched - * data file, the list of old non-equality delete files via {@code rewritable_delete_file_sets} - * ({@code TIcebergDeleteSink}/{@code TIcebergMergeSink}). The legacy fe-core path collected this from the - * {@code IcebergScanNode}'s own fields at sink-finalize time (plan-scoped, GC'd with the plan). In the plugin - * SPI the scan-plan provider and the write-plan provider are mutually-unaware, used-once objects; the only - * cross-scan→write survivor is the long-lived per-catalog {@link IcebergConnector}. So the scan provider stashes - * the supply here keyed by the statement's stable {@code queryId} ({@code ConnectorSession.getQueryId()} = - * {@code DebugUtil.printId(ctx.queryId())}, identical across the scan and write sessions of one statement), and - * the write provider retrieves it by the same key. - * - *

    Key = RAW data-file path. The map is keyed on the un-normalized {@code dataFile.path()} string - * ({@link IcebergScanRange#getOriginalPath()}), because the BE matches a rewritable set to the file it is writing - * a DV for by exact string equality against that raw path (mirrors legacy - * {@code IcebergScanNode.deleteFilesByReferencedDataFile} keyed on {@code getOriginalPath()}). Keying on the - * scheme-normalized open path would silently miss every lookup → resurrection. - * - *

    Eviction. The primary removal is the write provider's eager {@link #retrieveAndRemove} per statement - * (covers every DML write — DELETE/MERGE consume the supply, INSERT/OVERWRITE simply discard it). A statement - * that scans a v3-delete table but never writes (a plain {@code SELECT}, or a DML that errors before the write - * is planned) leaves a leaked entry that {@link #retrieveAndRemove} never reaches; a lazy TTL sweep (run when a - * new query is first seen) ages those out. Unlike a value cache, the sweep removes ONLY entries untouched for - * longer than the TTL — never a live entry (a statement's scan→write gap is milliseconds, far below the TTL), so - * a live supply is never dropped (which would itself resurrect rows). With a unique per-statement queryId a - * leaked entry is unreachable (no later statement reuses the key), so a leak is bounded memory, not a stale read. - */ -final class IcebergRewritableDeleteStash { - - // Leak backstop only: a statement's scan→write gap is milliseconds (both happen in one planning pass, before - // BE execution), so this TTL is orders of magnitude larger than any live entry's lifetime — it ages out only - // leaked plain-SELECT / aborted-DML entries. - private static final long DEFAULT_TTL_SECONDS = 300L; - - /** One in-flight statement's supply: rawDataFilePath -> its non-equality delete descs, plus a touch stamp. */ - private static final class Entry { - // Concurrent because, defensively, a single statement could plan more than one iceberg scan (MERGE scans - // both the target and the source table) — data-file paths are globally unique, so accumulating distinct - // keys (and idempotently overwriting a split data file's identical list) is always safe. - final Map> sets = new ConcurrentHashMap<>(); - volatile long lastTouchNanos; - - Entry(long nowNanos) { - this.lastTouchNanos = nowNanos; - } - } - - private final Map stash = new ConcurrentHashMap<>(); - private final long ttlNanos; - private final LongSupplier nanoClock; - - IcebergRewritableDeleteStash() { - this(DEFAULT_TTL_SECONDS, System::nanoTime); - } - - /** Visible for testing: injectable TTL + clock so sweep is deterministic without sleeping. */ - IcebergRewritableDeleteStash(long ttlSeconds, LongSupplier nanoClock) { - this.ttlNanos = TimeUnit.SECONDS.toNanos(Math.max(1L, ttlSeconds)); - this.nanoClock = nanoClock; - } - - /** - * Records, for {@code queryId}, the non-equality delete descs of one touched data file (keyed on its RAW - * path). No-op when the queryId is blank (a null/absent {@code ConnectContext} coerces the queryId to "", - * which would collide across concurrent statements — such a statement gets no supply, which is safe because a - * real row-level DML always carries a non-null query id) or when there is nothing to supply. Splits of one - * data file carry an identical delete list, so the per-path {@code put} is idempotent; two tables in one - * statement contribute distinct paths. - */ - void accumulate(String queryId, String rawDataFilePath, List deleteDescs) { - if (queryId == null || queryId.isEmpty() || rawDataFilePath == null - || deleteDescs == null || deleteDescs.isEmpty()) { - return; - } - long now = nanoClock.getAsLong(); - Entry entry = stash.get(queryId); - if (entry == null) { - // First range of a not-yet-seen query: opportunistically age out leaked entries. Done OUTSIDE the - // computeIfAbsent mapping function (ConcurrentHashMap forbids mutating other mappings from within it). - sweepExpired(now); - entry = stash.computeIfAbsent(queryId, k -> new Entry(now)); - } - entry.lastTouchNanos = now; - entry.sets.put(rawDataFilePath, deleteDescs); - } - - /** - * Returns and removes the supply map for {@code queryId} (the rewritable delete sets keyed by raw data-file - * path), or {@code null} when none was stashed / the queryId is blank. The write provider calls this once per - * statement; the remove is the primary eviction. - */ - Map> retrieveAndRemove(String queryId) { - if (queryId == null || queryId.isEmpty()) { - return null; - } - Entry entry = stash.remove(queryId); - return entry == null ? null : entry.sets; - } - - /** Drops entries untouched for longer than the TTL — leaked plain-SELECT / aborted-DML supplies only. */ - private void sweepExpired(long nowNanos) { - stash.entrySet().removeIf(e -> nowNanos - e.getValue().lastTouchNanos >= ttlNanos); - } - - /** Test-only: current number of in-flight stashed statements. */ - int size() { - return stash.size(); - } -} diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index e94cd0784a5cca..62fe48ff09cd5d 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -219,16 +219,10 @@ public class IcebergScanPlanProvider implements ConnectorScanPlanProvider { // Nullable — null via the 2-/3-arg ctors (offline tests, default-disabled gate); when null the gate is // forced off and planScan uses the SDK splitFiles path. private final IcebergManifestCache manifestCache; - // commit-bridge supply (S4 part 2): owned by the long-lived IcebergConnector, shared with the write provider. - // A format-version>=3 DELETE/MERGE scan stashes its non-equality delete supply here keyed by queryId; the - // write provider retrieves it to fill rewritable_delete_file_sets. Nullable — null via the 2-/3-/4-arg ctors - // (offline tests), in which case stashing is skipped (the supply is exercised only on the post-cutover write - // path; pre-flip the provider never runs at all). - private final IcebergRewritableDeleteStash rewritableDeleteStash; // PERF-01: cross-query RAW-table cache shared with the metadata layer, owned by the long-lived // IcebergConnector and injected via getScanPlanProvider. Nullable — null via the offline-test ctors and // when the connector's credential gate disables the cross-query layer; when null resolveTable still uses the - // query-scoped fat handle and falls back to a direct remote load. + // per-statement scope and falls back to a direct remote load. private final IcebergTableCache tableCache; // PERF-03: cross-query inferred-file-format cache shared with the connector, owned by the long-lived // IcebergConnector and injected via getScanPlanProvider. Nullable — null via the offline-test ctors; when null @@ -243,37 +237,30 @@ public class IcebergScanPlanProvider implements ConnectorScanPlanProvider { private final ConcurrentHashMap> scanProfileStash = new ConcurrentHashMap<>(); public IcebergScanPlanProvider(Map properties, IcebergCatalogOps catalogOps) { - this(properties, catalogOps, null, null, null); + this(properties, catalogOps, null, null); } public IcebergScanPlanProvider(Map properties, IcebergCatalogOps catalogOps, ConnectorContext context) { - this(properties, catalogOps, context, null, null); + this(properties, catalogOps, context, null); } public IcebergScanPlanProvider(Map properties, IcebergCatalogOps catalogOps, ConnectorContext context, IcebergManifestCache manifestCache) { - this(properties, catalogOps, context, manifestCache, null); - } - - public IcebergScanPlanProvider(Map properties, IcebergCatalogOps catalogOps, - ConnectorContext context, IcebergManifestCache manifestCache, - IcebergRewritableDeleteStash rewritableDeleteStash) { // Constant resolver: these ctors (offline tests + the pre-session connector paths) bind a single ops that // ignores the session, so existing behaviour/tests are byte-identical. No cross-query cache (tableCache - // null) — the fat handle still dedups within a planning pass. - this(properties, session -> catalogOps, context, manifestCache, rewritableDeleteStash, null); + // null) — the per-statement scope still dedups within a statement. + this(properties, session -> catalogOps, context, manifestCache, null); } /** * Session-aware convenience ctor without a cross-query table cache (tableCache null); used by the offline - * session-routing tests. The query-scoped fat handle still dedups within a planning pass. + * session-routing tests. The per-statement scope still dedups within a statement. */ public IcebergScanPlanProvider(Map properties, Function catalogOpsResolver, - ConnectorContext context, IcebergManifestCache manifestCache, - IcebergRewritableDeleteStash rewritableDeleteStash) { - this(properties, catalogOpsResolver, context, manifestCache, rewritableDeleteStash, null); + ConnectorContext context, IcebergManifestCache manifestCache) { + this(properties, catalogOpsResolver, context, manifestCache, null); } /** @@ -284,26 +271,23 @@ public IcebergScanPlanProvider(Map properties, */ public IcebergScanPlanProvider(Map properties, Function catalogOpsResolver, - ConnectorContext context, IcebergManifestCache manifestCache, - IcebergRewritableDeleteStash rewritableDeleteStash, IcebergTableCache tableCache) { - this(properties, catalogOpsResolver, context, manifestCache, rewritableDeleteStash, tableCache, null); + ConnectorContext context, IcebergManifestCache manifestCache, IcebergTableCache tableCache) { + this(properties, catalogOpsResolver, context, manifestCache, tableCache, null); } /** * Full ctor used by {@link IcebergConnector#getScanPlanProvider()}, adding the PERF-03 cross-query - * inferred-file-format cache ({@code formatCache}). The 6-arg ctor delegates here with a null format cache + * inferred-file-format cache ({@code formatCache}). The 5-arg ctor delegates here with a null format cache * (offline tests + pre-cache paths resolve {@code file_format_type} live). */ public IcebergScanPlanProvider(Map properties, Function catalogOpsResolver, - ConnectorContext context, IcebergManifestCache manifestCache, - IcebergRewritableDeleteStash rewritableDeleteStash, IcebergTableCache tableCache, + ConnectorContext context, IcebergManifestCache manifestCache, IcebergTableCache tableCache, IcebergFormatCache formatCache) { this.properties = properties; this.catalogOpsResolver = catalogOpsResolver; this.context = context; this.manifestCache = manifestCache; - this.rewritableDeleteStash = rewritableDeleteStash; this.tableCache = tableCache; this.formatCache = formatCache; } @@ -578,7 +562,7 @@ public boolean hasNext() { } while (iterator.hasNext()) { IcebergScanRange range = buildRangeForTask(iterator.next(), table, formatVersion, partitioned, - orderedPartitionKeys, zone, uriNormalizer, sliceSize, rewriteScope, false, null); + orderedPartitionKeys, zone, uriNormalizer, sliceSize, rewriteScope, null); if (range != null) { buffered = range; return true; @@ -669,15 +653,15 @@ private List planScanInternal( // and emit one BE-ready IcebergScanRange per task, populating the typed iceberg carriers — incl. the // merge-on-read delete files (T04) — mirroring legacy IcebergScanNode.createIcebergSplit. The field-id // history dict (T06, scan-level), MVCC pin, and vended credentials (T09) land later. - // commit-bridge supply (S4 part 2): for a format-version>=3 scan, stash each data file's non-equality - // delete supply (old DVs + old position deletes) keyed by the statement queryId, so a DELETE/MERGE write - // on the same statement can fill rewritable_delete_file_sets and the BE OR-merges those old deletes into - // the new deletion vector — a missing supply silently resurrects previously-deleted rows. queryId is read - // once (stable across this statement's scan and write sessions). Skipped pre-v3 and when the stash is - // absent (offline tests / pre-cutover the provider never runs); a non-DML scan just leaves a leaked entry - // the stash ages out, and accumulate() itself no-ops a blank queryId or an empty (no non-eq delete) list. - boolean stashRewritableDeletes = rewritableDeleteStash != null && formatVersion >= 3; - String stashQueryId = stashRewritableDeletes ? session.getQueryId() : null; + // commit-bridge supply (S4 part 2): for a format-version>=3 scan, accumulate each data file's non-equality + // delete supply (old DVs + old position deletes) into the per-statement scope, so a DELETE/MERGE write on + // the same statement can fill rewritable_delete_file_sets and the BE OR-merges those old deletes into the + // new deletion vector — a missing supply silently resurrects previously-deleted rows. The scope is keyed by + // catalog id + queryId (shared across this statement's scan and write, isolated per catalog for a + // cross-catalog MERGE). Skipped pre-v3; a non-DML scan just leaves an entry GC'd with the statement, and an + // absent scope (offline) yields a throwaway map that the write seam guards against (fail loud on v3 DML). + Map> rewritableDeleteSupply = formatVersion >= 3 + ? IcebergStatementScope.rewritableDeleteSupply(session) : null; // WS-REWRITE R2 per-group scope: when the handle carries a rewrite file scope (the engine // rewrite_data_files driver sets it before each group's INSERT-SELECT), keep ONLY the data files in @@ -696,7 +680,7 @@ private List planScanInternal( // identical to the streaming path's IcebergStreamingSplitSource so both produce the same ranges. IcebergScanRange range = buildRangeForTask(task, table, formatVersion, partitioned, orderedPartitionKeys, zone, uriNormalizer, plan.targetSplitSize, rewriteScope, - stashRewritableDeletes, stashQueryId); + rewritableDeleteSupply); if (range != null) { ranges.add(range); } @@ -747,15 +731,15 @@ private static String rootCauseMessage(Throwable t) { /** * Map one {@link FileScanTask} to its BE-ready {@link IcebergScanRange}, applying the rewrite-scope filter * (returns {@code null} to skip a data file outside the scope) and the v3 commit-bridge rewritable-delete - * stash side-effect. Shared by the synchronous {@link #planScanInternal} loop and the streaming + * accumulation. Shared by the synchronous {@link #planScanInternal} loop and the streaming * {@code IcebergStreamingSplitSource} so both paths produce byte-identical ranges and never drop a - * side-effect. The streaming path passes {@code stashRewritableDeletes=false} (v3 is gated onto the eager - * path — see {@link #streamingSplitEstimate}), so the stash is inert there. + * side-effect. The streaming path passes {@code rewritableDeleteSupply=null} (v3 is gated onto the eager + * path — see {@link #streamingSplitEstimate}), so the accumulation is inert there. */ private IcebergScanRange buildRangeForTask(FileScanTask task, Table table, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, UnaryOperator uriNormalizer, long targetSplitSize, Set rewriteScope, - boolean stashRewritableDeletes, String stashQueryId) { + Map> rewritableDeleteSupply) { DataFile dataFile = task.file(); if (rewriteScope != null && !rewriteScope.contains(dataFile.path().toString())) { return null; @@ -764,9 +748,14 @@ private IcebergScanRange buildRangeForTask(FileScanTask task, Table table, int f // size-proportional BE scheduling weight (selfSplitWeight computed inside buildRange). IcebergScanRange range = buildRange(table, dataFile, task, formatVersion, partitioned, orderedPartitionKeys, zone, uriNormalizer, -1, targetSplitSize); - if (stashRewritableDeletes) { - rewritableDeleteStash.accumulate(stashQueryId, range.getOriginalPath(), - range.rewritableDeleteDescs()); + if (rewritableDeleteSupply != null) { + // Record this data file's non-equality delete supply keyed on its RAW path (the exact string the BE + // matches a rewritable set against). Splits of one file carry an identical list, so the put is + // idempotent; an empty list (no old non-eq deletes) contributes nothing. + List descs = range.rewritableDeleteDescs(); + if (range.getOriginalPath() != null && descs != null && !descs.isEmpty()) { + rewritableDeleteSupply.put(range.getOriginalPath(), descs); + } } return range; } @@ -2274,26 +2263,22 @@ static ZoneId resolveSessionZone(ConnectorSession session) { * {@code null} context (offline unit tests / simple-auth) resolves directly. */ private Table resolveTable(ConnectorSession session, IcebergTableHandle handle) { - // Fat handle first (PERF-01): one handle threaded through getColumnHandles + planScan resolves the table - // once. The memo holds the RAW table; wrapTableForScan (the Kerberos doAs FileIO) is re-applied per call - // below so no per-request authenticator is ever frozen into the shared memo/cache. - Table cached = handle.getResolvedTable(); - if (cached != null) { - return wrapTableForScan(cached); - } - // Resolve the per-request ops before the auth scope so a session=user fail-closed surfaces verbatim. + // Per-statement scope (PERF-07): the statement's read metadata, scan planning and write all resolve the + // SAME one loaded RAW table. The scope holds the RAW table; wrapTableForScan (the Kerberos doAs FileIO) is + // re-applied per call below so no per-request authenticator is ever frozen into the shared object. + // Resolve the per-request ops before the auth scope so a session=user fail-closed surfaces verbatim (it + // re-validates the credential even on a scope hit). IcebergCatalogOps ops = catalogOpsResolver.apply(session); - Table raw; - if (context == null) { - raw = loadRawTable(ops, handle); - } else { + Table raw = IcebergStatementScope.sharedTable(session, handle.getDbName(), handle.getTableName(), () -> { + if (context == null) { + return loadRawTable(ops, handle); + } try { - raw = context.executeAuthenticated(() -> loadRawTable(ops, handle)); + return context.executeAuthenticated(() -> loadRawTable(ops, handle)); } catch (Exception e) { throw new RuntimeException("Failed to load table for scan, error message is:" + e.getMessage(), e); } - } - handle.setResolvedTable(raw); + }); return wrapTableForScan(raw); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java new file mode 100644 index 00000000000000..28b0e28fe41d49 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergStatementScope.java @@ -0,0 +1,84 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; +import org.apache.doris.thrift.TIcebergDeleteFileDesc; + +import org.apache.iceberg.Table; + +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Supplier; + +/** + * Connector-private helpers over the neutral {@link ConnectorStatementScope} (reached via + * {@link ConnectorSession#getStatementScope()}), giving iceberg one place to key its per-statement state. + * + *

    The scope is the per-statement table-load owner: the read metadata path, scan planning, write shaping + * and {@code beginWrite} all resolve one table through {@link #sharedTable} so a single statement loads each + * table once and every resolver shares that one RAW object (snapshot pins and auth wraps are applied per + * consumer, never frozen into the shared object). It also carries the merge-on-read rewritable-delete supply + * from the scan seam to the write seam ({@link #rewritableDeleteSupply}), replacing the former per-catalog + * singleton stash — the scope is per-statement, so a statement's supply is GC'd with it and a reused + * prepared-statement scope is reset per execution (see {@code ExecuteCommand}).

    + * + *

    Under {@link ConnectorStatementScope#NONE} (offline planning / no live statement) {@link #sharedTable} + * loads every time (byte-identical to the pre-scope behavior) and {@link #rewritableDeleteSupply} returns a + * throwaway map that does NOT bridge scan→write — so a format-version≥3 row-level DML under NONE fails + * loud at the write seam rather than silently resurrecting rows.

    + */ +final class IcebergStatementScope { + + private IcebergStatementScope() {} + + /** + * Loads the RAW iceberg {@link Table} for {@code db.tbl} once per statement and shares it across every + * resolver. The key includes the catalog id (cross-catalog MERGE isolation) and the statement's queryId + * (a reused prepared context sees each execution's own table). {@code loader} runs at most once per + * statement — callers pass the raw load (cross-query cache or direct remote) and own the auth scope + * (the caller wraps this in {@code executeAuthenticated}). + */ + static Table sharedTable(ConnectorSession session, String dbName, String tableName, Supplier
    loader) { + if (session == null) { + // No session (offline / direct-construction tests): load every time, like ConnectorStatementScope.NONE. + return loader.get(); + } + String key = "iceberg.table:" + session.getCatalogId() + ":" + dbName + ":" + tableName + + ":" + session.getQueryId(); + return session.getStatementScope().computeIfAbsent(key, loader); + } + + /** + * Returns this statement's rewritable-delete supply map (RAW data-file path → its non-equality delete + * descs), creating it empty on first use. The scan seam accumulates into it (per touched data file) and the + * write seam drains it; keyed by catalog id + queryId so a cross-catalog MERGE keeps each table's supply + * isolated. Under {@link ConnectorStatementScope#NONE} each call returns a fresh throwaway map, so scan and + * write do NOT share — the write seam guards format-version≥3 DML against that (fail loud). + */ + static Map> rewritableDeleteSupply(ConnectorSession session) { + if (session == null) { + // No session: a throwaway map that does NOT bridge scan->write (same as NONE). + return new ConcurrentHashMap<>(); + } + String key = "iceberg.rewritable-delete-supply:" + session.getCatalogId() + ":" + session.getQueryId(); + return session.getStatementScope().computeIfAbsent(key, ConcurrentHashMap::new); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableHandle.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableHandle.java index 8872f00dc0aaf2..295402a0f66de9 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableHandle.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTableHandle.java @@ -20,7 +20,6 @@ import org.apache.doris.connector.api.handle.ConnectorTableHandle; import com.google.common.collect.ImmutableSet; -import org.apache.iceberg.Table; import java.util.Objects; import java.util.Set; @@ -97,21 +96,6 @@ public class IcebergTableHandle implements ConnectorTableHandle { */ private final boolean topnLazyMaterialize; - /** - * Query-scoped resolved {@link Table} memo (PERF-01): the RAW iceberg table this handle resolves to, cached - * so the many read entries that share one handle within a single planning/analysis pass - * ({@code getColumnHandles}, {@code getTableStatistics}, the scan provider's {@code resolveTable}, ...) - * collapse their remote {@code loadTable} RPCs onto one. It is {@code transient} — NOT part of the handle's - * serialized form (BE never sees it) — and NOT part of the handle identity: {@link #equals}/ - * {@link #hashCode}/{@link #toString} deliberately ignore it, since it is a resolution cache, not a - * coordinate. The {@code with*} copies carry it forward (an iceberg branch/tag/time-travel pin is the SAME - * base {@code Table} plus a downstream {@code useSnapshot}/{@code useRef}, and {@code getColumnHandles} - * resolves the table BEFORE the snapshot pin is threaded on), so a pin copy must not force a re-load. A - * deserialized handle (BE side) starts {@code null} and resolves live. The stored table is RAW: the scan - * provider applies {@code wrapTableForScan} (Kerberos FileIO) per call, never freezing it into the memo. - */ - private transient Table resolvedTable; - public IcebergTableHandle(String dbName, String tableName) { this(dbName, tableName, NO_PIN, null, NO_PIN, null, null, false); } @@ -191,16 +175,6 @@ public boolean isTopnLazyMaterialize() { return topnLazyMaterialize; } - /** The query-scoped resolved RAW table memo, or {@code null} if not yet resolved (see {@link #resolvedTable}). */ - public Table getResolvedTable() { - return resolvedTable; - } - - /** Memoizes the resolved RAW table on this handle so later reads sharing it skip the remote load. */ - public void setResolvedTable(Table resolvedTable) { - this.resolvedTable = resolvedTable; - } - /** * Returns a copy of this handle carrying the resolved time-travel pin. Mirrors paimon's * {@code PaimonTableHandle.withScanOptions}/{@code withBranch} but with iceberg's typed carriers. @@ -209,13 +183,8 @@ public IcebergTableHandle withSnapshot(long snapshotId, String ref, long schemaI // sysTableName, rewriteFileScope and topnLazyMaterialize are preserved: threading a resolved // time-travel pin in must not degrade a sys handle (t$snapshots) into a normal data-table handle, // drop a rewrite scope, or drop the lazy-materialization signal. - IcebergTableHandle copy = new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, + return new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, rewriteFileScope, topnLazyMaterialize); - // Carry the resolved-table memo forward: the pin is applied downstream via useSnapshot/useRef on the - // SAME base table, and getColumnHandles resolves the table before applySnapshot copies the handle, so a - // pin copy must not drop the memo and force a re-load (PERF-01). - copy.resolvedTable = resolvedTable; - return copy; } /** @@ -227,10 +196,8 @@ public IcebergTableHandle withSnapshot(long snapshotId, String ref, long schemaI * The other carriers (snapshot/ref/schema/sys) are preserved. */ public IcebergTableHandle withRewriteFileScope(Set rawDataFilePaths) { - IcebergTableHandle copy = new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, + return new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, ImmutableSet.copyOf(rawDataFilePaths), topnLazyMaterialize); - copy.resolvedTable = resolvedTable; - return copy; } /** @@ -238,10 +205,8 @@ public IcebergTableHandle withRewriteFileScope(Set rawDataFilePaths) { * {@link #topnLazyMaterialize}). The other carriers (snapshot/ref/schema/sys/rewriteScope) are preserved. */ public IcebergTableHandle withTopnLazyMaterialize(boolean topnLazyMaterialize) { - IcebergTableHandle copy = new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, + return new IcebergTableHandle(dbName, tableName, snapshotId, ref, schemaId, sysTableName, rewriteFileScope, topnLazyMaterialize); - copy.resolvedTable = resolvedTable; - return copy; } @Override diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java index aaa3e1ff050c76..4ad5d97c5fd9a6 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWritePlanProvider.java @@ -19,6 +19,7 @@ import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; @@ -91,7 +92,7 @@ * write distribution and the vended-credentials overlay of the hadoop config are registered deviations * (DV-T0x-vended / -broker / -materialize) closed at the P6.6 cutover. At format-version≥3 the DELETE / * MERGE sink's {@code rewritable_delete_file_sets} is filled here from the scan-time supply the - * {@link IcebergRewritableDeleteStash} carried across the scan→write seam (commit-bridge S4 part 2), + * per-statement {@link IcebergStatementScope} carried across the scan→write seam (commit-bridge S4 part 2), * replacing the legacy fe-resident rewritable-delete planner.

    * *

    Gate-closed / dormant. Iceberg is not in {@code SPI_READY_TYPES} until P6.6, so nothing @@ -137,22 +138,12 @@ private static ConnectorColumn buildRowIdColumn() { // single shared ops regardless of session (constant s -> catalogOps). private final Function catalogOpsResolver; private final ConnectorContext context; - // commit-bridge supply (S4 part 2): the per-catalog stash the scan provider filled with each touched data - // file's non-equality delete supply. planWrite retrieves (and evicts) it by queryId to fill the v3 - // rewritable_delete_file_sets. Nullable — null via the 3-arg ctor (offline tests), in which case no supply is - // attached (and the BE would resurrect rows, which is why the cutover wiring injects the real stash). - private final IcebergRewritableDeleteStash rewritableDeleteStash; public IcebergWritePlanProvider(Map properties, IcebergCatalogOps catalogOps, ConnectorContext context) { - this(properties, catalogOps, context, null); - } - - public IcebergWritePlanProvider(Map properties, IcebergCatalogOps catalogOps, - ConnectorContext context, IcebergRewritableDeleteStash rewritableDeleteStash) { // Constant resolver: these ctors (offline tests) bind a single ops that ignores the session, so existing // behaviour/tests are byte-identical. - this(properties, session -> catalogOps, context, rewritableDeleteStash); + this(properties, session -> catalogOps, context); } /** @@ -163,11 +154,10 @@ public IcebergWritePlanProvider(Map properties, IcebergCatalogOp */ public IcebergWritePlanProvider(Map properties, Function catalogOpsResolver, - ConnectorContext context, IcebergRewritableDeleteStash rewritableDeleteStash) { + ConnectorContext context) { this.properties = properties; this.catalogOpsResolver = catalogOpsResolver; this.context = context; - this.rewritableDeleteStash = rewritableDeleteStash; } @Override @@ -183,13 +173,24 @@ public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandl transaction.beginWrite(session, tableHandle.getDbName(), tableHandle.getTableName(), writeContext); Table table = transaction.getTable(); - // commit-bridge supply (S4 part 2): retrieve (and evict) the non-equality delete supply the scan provider - // stashed for this statement. Done once for every write op — DELETE/MERGE attach it to the sink so the BE - // OR-merges old deletes into the new deletion vector (a missing supply silently resurrects deleted rows); - // INSERT/OVERWRITE discard it, but the retrieve still evicts the stash entry (e.g. an INSERT ... SELECT - // FROM an iceberg source). Null when the stash is absent (offline tests) or nothing was stashed. - Map> rewritableDeletes = rewritableDeleteStash != null - ? rewritableDeleteStash.retrieveAndRemove(session.getQueryId()) : null; + // commit-bridge supply (S4 part 2): read the non-equality delete supply the scan seam accumulated into the + // per-statement scope. DELETE/MERGE attach it to the sink so the BE OR-merges old deletes into the new + // deletion vector (a missing supply silently resurrects deleted rows); INSERT/OVERWRITE/REWRITE ignore it + // (buildRewritableDeleteFileSets no-ops an empty map, same as the former null). + // Fail loud (never silently resurrect): a format-version>=3 row-level DML under an absent statement scope + // (ConnectorStatementScope.NONE — offline / no live statement) cannot have received the scan's supply (each + // NONE lookup is a throwaway map), so reject it rather than write a deletion vector that drops old deletes. + WriteOperation writeOp = writeContext.getWriteOperation(); + if ((writeOp == WriteOperation.DELETE || writeOp == WriteOperation.UPDATE || writeOp == WriteOperation.MERGE) + && session.getStatementScope() == ConnectorStatementScope.NONE + && IcebergWriterHelper.getFormatVersion(table) >= 3) { + throw new DorisConnectorException("Iceberg row-level " + writeOp + " on a format-version>=3 table requires " + + "a per-statement scope to carry the rewritable-delete supply from scan to write; none is present " + + "(ConnectorStatementScope.NONE). Refusing to write a deletion vector that would drop old deletes " + + "and resurrect previously-deleted rows."); + } + Map> rewritableDeletes = + IcebergStatementScope.rewritableDeleteSupply(session); // Dispatch on the write operation to the matching BE sink dialect (each is a distinct TDataSinkType, // byte-identical to the legacy fe-core planner sink). OVERWRITE shares TIcebergTableSink with INSERT @@ -687,18 +688,24 @@ private IcebergConnectorTransaction currentTransaction(ConnectorSession session) } private Table resolveTable(ConnectorSession session, IcebergTableHandle handle) { - // Resolve the per-request ops before the auth scope so a session=user fail-closed surfaces verbatim. + // Per-statement scope (PERF-07): the write-shaping derivations (sort columns / partitioning / explain) + // share the SAME one loaded RAW table as the statement's read + scan resolvers. For a row-level DML the + // scan has already populated the scope, so this is a hit (no write-side load); a write-only INSERT loads + // once here. Resolve the per-request ops before the auth scope so a session=user fail-closed surfaces + // verbatim (it re-validates the credential even on a scope hit). IcebergCatalogOps ops = catalogOpsResolver.apply(session); - if (context == null) { - return ops.loadTable(handle.getDbName(), handle.getTableName()); - } - try { - return context.executeAuthenticated( - () -> ops.loadTable(handle.getDbName(), handle.getTableName())); - } catch (Exception e) { - throw new DorisConnectorException("Failed to load iceberg table " - + handle.getDbName() + "." + handle.getTableName() + ": " + e.getMessage(), e); - } + return IcebergStatementScope.sharedTable(session, handle.getDbName(), handle.getTableName(), () -> { + if (context == null) { + return ops.loadTable(handle.getDbName(), handle.getTableName()); + } + try { + return context.executeAuthenticated( + () -> ops.loadTable(handle.getDbName(), handle.getTableName())); + } catch (Exception e) { + throw new DorisConnectorException("Failed to load iceberg table " + + handle.getDbName() + "." + handle.getTableName() + ": " + e.getMessage(), e); + } + }); } private static TFileFormatType toTFileFormatType(FileFormat format) { diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergProviderSessionRoutingTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergProviderSessionRoutingTest.java index 632a112f23dbdf..319bebec8600c5 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergProviderSessionRoutingTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergProviderSessionRoutingTest.java @@ -108,7 +108,7 @@ public void scanProviderRoutesCredentialedSessionToPerUserOps() { public void writeProviderAppliesResolverWithCallSessionAndFailsClosed() { List seen = new ArrayList<>(); IcebergWritePlanProvider provider = new IcebergWritePlanProvider(Collections.emptyMap(), - failClosedResolver(seen, new RecordingIcebergCatalogOps()), null, null); + failClosedResolver(seen, new RecordingIcebergCatalogOps()), null); RoutingSession noCred = new RoutingSession(null); DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStashTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStashTest.java deleted file mode 100644 index 8f1b40dff9a87f..00000000000000 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergRewritableDeleteStashTest.java +++ /dev/null @@ -1,204 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.connector.iceberg; - -import org.apache.doris.thrift.TIcebergDeleteFileDesc; - -import org.junit.jupiter.api.Assertions; -import org.junit.jupiter.api.Test; - -import java.util.Arrays; -import java.util.Collections; -import java.util.List; -import java.util.Map; -import java.util.concurrent.TimeUnit; -import java.util.concurrent.atomic.AtomicLong; - -/** - * Unit-pins {@link IcebergRewritableDeleteStash}, the scan→write seam carrier for a row-level DML's - * non-equality delete supply. WHY each invariant matters: the BE OR-merges the supplied old deletes into the - * new deletion vector, so a DROPPED or WRONG supply silently resurrects previously-deleted rows; an over-stash - * (a plain SELECT that never writes) must NOT grow unboundedly. The clock is injected so the TTL sweep is - * deterministic without sleeping. - */ -public class IcebergRewritableDeleteStashTest { - - private static TIcebergDeleteFileDesc desc(String path, int content) { - TIcebergDeleteFileDesc d = new TIcebergDeleteFileDesc(); - d.setPath(path); - d.setContent(content); - return d; - } - - private static List descs(String path, int content) { - return Collections.singletonList(desc(path, content)); - } - - /** A stash whose clock the test drives by hand (nanos), so TTL expiry is exact. */ - private static IcebergRewritableDeleteStash stashWithClock(long ttlSeconds, AtomicLong nanos) { - return new IcebergRewritableDeleteStash(ttlSeconds, nanos::get); - } - - @Test - public void retrieveReturnsWhatWasAccumulatedKeyedByRawDataFilePath() { - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate("q1", "s3://b/db/t/f1.parquet", descs("s3://b/db/t/dv1.puffin", 3)); - - Map> sets = stash.retrieveAndRemove("q1"); - Assertions.assertNotNull(sets); - Assertions.assertEquals(1, sets.size()); - // The KEY is the raw data-file path (the string the BE matches against). A mutation keying on anything - // else loses the lookup -> resurrection. - Assertions.assertTrue(sets.containsKey("s3://b/db/t/f1.parquet")); - Assertions.assertEquals("s3://b/db/t/dv1.puffin", sets.get("s3://b/db/t/f1.parquet").get(0).getPath()); - } - - @Test - public void retrieveAndRemoveEvictsSoASecondRetrieveIsEmpty() { - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate("q1", "s3://b/db/t/f1.parquet", descs("dv", 3)); - - Assertions.assertNotNull(stash.retrieveAndRemove("q1")); - // The retrieve is the primary eviction. MUTATION: making retrieve NOT remove -> this second read is - // non-null and the entry leaks across statements. - Assertions.assertNull(stash.retrieveAndRemove("q1")); - Assertions.assertEquals(0, stash.size()); - } - - @Test - public void retrieveUnknownQueryIdReturnsNull() { - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - Assertions.assertNull(stash.retrieveAndRemove("never-stashed")); - } - - @Test - public void accumulateMergesDistinctDataFilesUnderOneQuery() { - // A MERGE scans two tables under one queryId; their data-file paths are globally distinct, so both must - // survive in one entry. MUTATION: overwriting (put under the queryId) instead of merging per path drops - // the first table's supply -> resurrection on that table. - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate("q1", "s3://b/db/target/f.parquet", descs("dv-target", 3)); - stash.accumulate("q1", "s3://b/db/source/g.parquet", descs("dv-source", 3)); - - Map> sets = stash.retrieveAndRemove("q1"); - Assertions.assertEquals(2, sets.size()); - Assertions.assertTrue(sets.containsKey("s3://b/db/target/f.parquet")); - Assertions.assertTrue(sets.containsKey("s3://b/db/source/g.parquet")); - } - - @Test - public void accumulateSamePathIsIdempotentForSplitRanges() { - // A large data file split into several ranges carries an identical delete list per split; re-putting the - // same key must not duplicate. Last-wins (same value) keeps exactly one entry. - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - List d = descs("dv", 3); - stash.accumulate("q1", "s3://b/db/t/f.parquet", d); - stash.accumulate("q1", "s3://b/db/t/f.parquet", d); - - Map> sets = stash.retrieveAndRemove("q1"); - Assertions.assertEquals(1, sets.size()); - Assertions.assertEquals(1, sets.get("s3://b/db/t/f.parquet").size()); - } - - @Test - public void twoQueryIdsAreIsolated() { - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate("q1", "s3://b/db/t/f1.parquet", descs("dv1", 3)); - stash.accumulate("q2", "s3://b/db/t/f2.parquet", descs("dv2", 3)); - - Assertions.assertTrue(stash.retrieveAndRemove("q1").containsKey("s3://b/db/t/f1.parquet")); - // q1's retrieve must not touch q2. - Assertions.assertTrue(stash.retrieveAndRemove("q2").containsKey("s3://b/db/t/f2.parquet")); - } - - @Test - public void blankQueryIdIsNeverStashed() { - // A null/absent ConnectContext coerces queryId to "" — two concurrent such statements would collide on - // "" and read each other's (or stale) supply. MUTATION: dropping the blank guard lets the "" key store a - // map that a later null-ctx statement reads -> stale supply / resurrection. - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate("", "s3://b/db/t/f.parquet", descs("dv", 3)); - stash.accumulate(null, "s3://b/db/t/f.parquet", descs("dv", 3)); - - Assertions.assertEquals(0, stash.size()); - Assertions.assertNull(stash.retrieveAndRemove("")); - Assertions.assertNull(stash.retrieveAndRemove(null)); - } - - @Test - public void emptyOrNullSupplyIsNotStashed() { - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate("q1", "s3://b/db/t/f.parquet", Collections.emptyList()); - stash.accumulate("q1", "s3://b/db/t/f.parquet", null); - - Assertions.assertEquals(0, stash.size()); - } - - @Test - public void ttlSweepEvictsLeakedEntryOnceExpiredButNotBefore() { - AtomicLong nanos = new AtomicLong(0L); - IcebergRewritableDeleteStash stash = stashWithClock(300L, nanos); - - // q1 scanned but never writes (a leaked plain-SELECT entry). - stash.accumulate("q1", "s3://b/db/t/f1.parquet", descs("dv1", 3)); - Assertions.assertEquals(1, stash.size()); - - // 299s later: a new query arrives; q1 is still within TTL so it is NOT swept (it could still be a live - // supply whose write is pending). MUTATION: sweeping a live entry here would resurrect its rows. - nanos.set(TimeUnit.SECONDS.toNanos(299L)); - stash.accumulate("q2", "s3://b/db/t/f2.parquet", descs("dv2", 3)); - Assertions.assertEquals(2, stash.size()); - - // Past the TTL: the next new-query accumulate sweeps the now-expired q1 (but keeps the fresh q3). - nanos.set(TimeUnit.SECONDS.toNanos(301L)); - stash.accumulate("q3", "s3://b/db/t/f3.parquet", descs("dv3", 3)); - Assertions.assertNull(stash.retrieveAndRemove("q1")); - Assertions.assertNotNull(stash.retrieveAndRemove("q2")); - Assertions.assertNotNull(stash.retrieveAndRemove("q3")); - } - - @Test - public void accumulateRefreshesTouchStampSoAnActiveQueryIsNotSwept() { - AtomicLong nanos = new AtomicLong(0L); - IcebergRewritableDeleteStash stash = stashWithClock(300L, nanos); - - stash.accumulate("q1", "s3://b/db/t/f1.parquet", descs("dv1", 3)); - // q1 keeps adding ranges over time (a long scan). At 200s a fresh range refreshes its stamp. - nanos.set(TimeUnit.SECONDS.toNanos(200L)); - stash.accumulate("q1", "s3://b/db/t/f2.parquet", descs("dv2", 3)); - // 400s absolute = 200s since the last touch: still within TTL, must survive a sweep triggered by q9. - nanos.set(TimeUnit.SECONDS.toNanos(400L)); - stash.accumulate("q9", "s3://b/db/t/g.parquet", descs("dvg", 3)); - - Map> sets = stash.retrieveAndRemove("q1"); - Assertions.assertNotNull(sets); - Assertions.assertEquals(2, sets.size()); - } - - @Test - public void multipleDescsForOneDataFileAreAllCarried() { - // A v2->v3 upgraded data file can have BOTH an old position-delete file and an old DV; both must reach - // the BE for the union, or the rows covered by the missing one resurrect. - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate("q1", "s3://b/db/t/f.parquet", - Arrays.asList(desc("pos.parquet", 1), desc("dv.puffin", 3))); - - Map> sets = stash.retrieveAndRemove("q1"); - Assertions.assertEquals(2, sets.get("s3://b/db/t/f.parquet").size()); - } -} diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index 56d5e2b6af7e40..28abf7eed93f66 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.iceberg; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; @@ -230,26 +231,46 @@ public void planScanResolvesTableInsideAuthContext() { } @Test - public void planningPassLoadsSameTableOnceViaFatHandle() { - // PERF-01 metric gate: a single planning pass threads ONE IcebergTableHandle through getColumnHandles - // (metadata) and planScan (provider). The fat-handle memo (transient resolvedTable set by the first - // resolve, reused by the second) collapses BOTH remote reads onto a single loadTable RPC. This is the - // deterministic core claim — one remote loadTable per table per planning pass, independent of the - // cross-query cache (off here: context-less provider, disabled metadata cache). - // MUTATION: dropping the fat handle (each resolve re-loads) -> 2 loadTable log entries -> red. + public void planningPassLoadsSameTableOnceViaSharedScope() { + // PERF-07 metric gate: a statement's metadata read (getColumnHandles) and scan planning (planScan) resolve + // the SAME table. Sharing ONE per-statement scope across both (same session) collapses BOTH remote reads + // onto a single loadTable RPC. This is the deterministic core claim — one remote loadTable per table per + // statement, independent of the cross-query cache (off here: disabled metadata cache, null-cache provider). + // MUTATION: not routing through the scope (each resolve re-loads) -> 2 loadTable log entries -> red. Table empty = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); RecordingIcebergCatalogOps ops = opsReturning(empty); IcebergConnectorMetadata metadata = new IcebergConnectorMetadata(ops, Collections.emptyMap(), new RecordingConnectorContext()); IcebergScanPlanProvider provider = new IcebergScanPlanProvider(Collections.emptyMap(), ops); IcebergTableHandle handle = new IcebergTableHandle("db1", "t1"); + ConnectorSession session = new FakeScanSession("UTC", Collections.emptyMap()) + .withScope(new TestStatementScope()); - metadata.getColumnHandles(null, handle); - provider.planScan(null, handle, Collections.emptyList(), Optional.empty()); + metadata.getColumnHandles(session, handle); + provider.planScan(session, handle, Collections.emptyList(), Optional.empty()); long remoteLoads = ops.log.stream().filter("loadTable:db1.t1"::equals).count(); Assertions.assertEquals(1, remoteLoads, - "fat handle must collapse the metadata + provider reads to one remote loadTable"); + "the per-statement scope must collapse the metadata + provider reads to one remote loadTable"); + } + + @Test + public void planningPassWithoutSharedScopeLoadsEachTime() { + // Contrast to the shared-scope gate: with NONE (no live statement scope) each resolver loads independently + // (byte-identical to the pre-scope offline behavior). MUTATION: memoizing under NONE -> 1 load -> red. + Table empty = createTable("t1", SCHEMA, PartitionSpec.unpartitioned()); + RecordingIcebergCatalogOps ops = opsReturning(empty); + IcebergConnectorMetadata metadata = + new IcebergConnectorMetadata(ops, Collections.emptyMap(), new RecordingConnectorContext()); + IcebergScanPlanProvider provider = new IcebergScanPlanProvider(Collections.emptyMap(), ops); + IcebergTableHandle handle = new IcebergTableHandle("db1", "t1"); + ConnectorSession none = new FakeScanSession("UTC", Collections.emptyMap()); + + metadata.getColumnHandles(none, handle); + provider.planScan(none, handle, Collections.emptyList(), Optional.empty()); + + long remoteLoads = ops.log.stream().filter("loadTable:db1.t1"::equals).count(); + Assertions.assertEquals(2, remoteLoads, "under NONE each resolver loads (no memo)"); } // --- T02 split-enumeration + predicate-pushdown tests --- @@ -1185,10 +1206,11 @@ public void planScanReadsRealFormatVersionAndEmitsV3RowLineage() { // ── commit-bridge supply (S4 part 2): a v3 scan stashes each data file's non-equality deletes by raw path ── @Test - public void planScanStashesRewritableDeletesKeyedByRawDataFilePathForV3() { - // A v3 scan over a data file that already has a deletion vector must stash that DV keyed on the data - // file's RAW path, so a same-statement DELETE/MERGE write can hand it to the BE. MUTATION: not stashing - // (or keying on the normalized path) -> the write supplies nothing -> the BE resurrects the deleted rows. + public void planScanAccumulatesRewritableDeletesKeyedByRawDataFilePathForV3() { + // A v3 scan over a data file that already has a deletion vector must accumulate that DV into the + // per-statement scope keyed on the data file's RAW path, so a same-statement DELETE/MERGE write can hand + // it to the BE. MUTATION: not accumulating (or keying on the normalized path) -> the write supplies + // nothing -> the BE resurrects the deleted rows. Map v3 = new HashMap<>(); v3.put("format-version", "3"); Table table = createTable("v3dv", SCHEMA, PartitionSpec.unpartitioned(), v3); @@ -1199,27 +1221,26 @@ public void planScanStashesRewritableDeletesKeyedByRawDataFilePathForV3() { .addDeletes(deletionVectorFile("s3://b/db/t1/dv.puffin", 16L, 64L)) .commit(); - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - IcebergScanPlanProvider provider = - new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table), null, null, stash); - provider.planScan(new FakeScanSession("UTC", Collections.emptyMap()), - new IcebergTableHandle("db1", "v3dv"), Collections.emptyList(), Optional.empty()); + IcebergScanPlanProvider provider = new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table)); + ConnectorSession session = new FakeScanSession("UTC", Collections.emptyMap()) + .withScope(new TestStatementScope()); + provider.planScan(session, new IcebergTableHandle("db1", "v3dv"), Collections.emptyList(), Optional.empty()); - Map> sets = stash.retrieveAndRemove("q"); - Assertions.assertNotNull(sets, "a v3 scan with a live DV must stash a supply for queryId 'q'"); + Map> sets = IcebergStatementScope.rewritableDeleteSupply(session); + Assertions.assertFalse(sets.isEmpty(), "a v3 scan with a live DV must accumulate a supply into the scope"); // Keyed on the RAW data-file path (== originalPath), the string the BE matches a rewritable set against. Assertions.assertTrue(sets.containsKey("s3://b/db/t1/f1.parquet"), - "stash must key on the raw data-file path, got keys: " + sets.keySet()); + "supply must key on the raw data-file path, got keys: " + sets.keySet()); List descs = sets.get("s3://b/db/t1/f1.parquet"); Assertions.assertEquals(1, descs.size()); Assertions.assertEquals(3, descs.get(0).getContent(), "the DV is content 3"); } @Test - public void planScanDoesNotStashForVersionTwo() { + public void planScanDoesNotAccumulateForVersionTwo() { // v2 deletes are plain position-delete files (no DV union); the rewritable supply is a v3-only concept. // A real position delete is committed so the assertion proves the formatVersion>=3 GATE, not an absence - // of deletes. MUTATION: dropping the v3 gate -> this v2 position delete would be stashed -> red. + // of deletes. MUTATION: dropping the v3 gate -> this v2 position delete would be accumulated -> red. Table table = createTable("v2pd", SCHEMA, PartitionSpec.unpartitioned(), Collections.singletonMap("format-version", "2")); table.newAppend() @@ -1229,18 +1250,19 @@ public void planScanDoesNotStashForVersionTwo() { .addDeletes(positionDeleteFile("s3://b/db/t1/pos.parquet", FileFormat.PARQUET, null, null)) .commit(); - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - IcebergScanPlanProvider provider = - new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table), null, null, stash); - provider.planScan(new FakeScanSession("UTC", Collections.emptyMap()), - new IcebergTableHandle("db1", "v2pd"), Collections.emptyList(), Optional.empty()); + IcebergScanPlanProvider provider = new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table)); + ConnectorSession session = new FakeScanSession("UTC", Collections.emptyMap()) + .withScope(new TestStatementScope()); + provider.planScan(session, new IcebergTableHandle("db1", "v2pd"), Collections.emptyList(), Optional.empty()); - Assertions.assertEquals(0, stash.size(), "a v2 scan must not stash any rewritable supply"); + Assertions.assertTrue(IcebergStatementScope.rewritableDeleteSupply(session).isEmpty(), + "a v2 scan must not accumulate any rewritable supply"); } @Test - public void planScanWithoutStashIsInert() { - // The offline 2-arg ctor leaves the stash null; a v3 scan must not NPE — it simply skips stashing. + public void planScanUnderNoneScopeIsInert() { + // Under a NONE scope (offline / no live statement) a v3 scan must not NPE — it simply accumulates into a + // throwaway map that does not bridge to any write. MUTATION: dereferencing a missing scope -> NPE -> red. Map v3 = new HashMap<>(); v3.put("format-version", "3"); Table table = createTable("v3ns", SCHEMA, PartitionSpec.unpartitioned(), v3); @@ -1449,12 +1471,24 @@ public void streamSplitsCloseBeforeIterationDoesNotThrow() throws IOException { private static final class FakeScanSession implements ConnectorSession { private final String timeZone; private final Map sessionProperties; + private ConnectorStatementScope statementScope = ConnectorStatementScope.NONE; FakeScanSession(String timeZone, Map sessionProperties) { this.timeZone = timeZone; this.sessionProperties = sessionProperties; } + /** Installs a memoizing per-statement scope; share one instance across sessions to mimic one statement. */ + FakeScanSession withScope(ConnectorStatementScope scope) { + this.statementScope = scope; + return this; + } + + @Override + public ConnectorStatementScope getStatementScope() { + return statementScope; + } + @Override public String getQueryId() { return "q"; diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergStatementScopeTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergStatementScopeTest.java new file mode 100644 index 00000000000000..70238ea323f82d --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergStatementScopeTest.java @@ -0,0 +1,179 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; +import org.apache.doris.thrift.TIcebergDeleteFileDesc; + +import org.apache.iceberg.PartitionSpec; +import org.apache.iceberg.Schema; +import org.apache.iceberg.Table; +import org.apache.iceberg.types.Types; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Tests for {@link IcebergStatementScope}: the per-statement table memo keying (catalog id + db + table + + * queryId) that lets a statement's read/scan/write share ONE loaded table, and the rewritable-delete supply + * map keying (catalog id + queryId) that bridges the scan→write seam. + */ +public class IcebergStatementScopeTest { + + private static final Schema SCHEMA = + new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get())); + + private static Table table(String name) { + return new FakeIcebergTable(name, SCHEMA, PartitionSpec.unpartitioned(), + "s3://b/db1/" + name, Collections.emptyMap()); + } + + @Test + public void sameStatementSharesOneLoadedTable() { + // The read, scan and write resolvers of one statement resolve the same table through sharedTable; sharing + // one scope collapses them onto one load and hands each the SAME instance (read/write share). + // MUTATION: not memoizing -> two loads / two instances -> red. + ScopeSession session = new ScopeSession(7L, "q1", new TestStatementScope()); + AtomicInteger loads = new AtomicInteger(); + Table t1 = IcebergStatementScope.sharedTable(session, "db1", "t", () -> { + loads.incrementAndGet(); + return table("t"); + }); + Table t2 = IcebergStatementScope.sharedTable(session, "db1", "t", () -> { + loads.incrementAndGet(); + return table("t"); + }); + Assertions.assertSame(t1, t2, "same statement + table -> one shared instance"); + Assertions.assertEquals(1, loads.get(), "loaded once per statement"); + } + + @Test + public void differentQueryIdIsolatesTheLoad() { + // A reused prepared statement runs each EXECUTE under its own queryId, so one execution never sees + // another's table even on the same scope object. + TestStatementScope scope = new TestStatementScope(); + Table a = IcebergStatementScope.sharedTable(new ScopeSession(7L, "q1", scope), "db1", "t", () -> table("t")); + Table b = IcebergStatementScope.sharedTable(new ScopeSession(7L, "q2", scope), "db1", "t", () -> table("t")); + Assertions.assertNotSame(a, b, "different queryId -> isolated load"); + } + + @Test + public void differentCatalogIdIsolatesTheLoad() { + // A cross-catalog MERGE resolves the two catalogs' tables independently (the key carries the catalog id). + TestStatementScope scope = new TestStatementScope(); + Table a = IcebergStatementScope.sharedTable(new ScopeSession(1L, "q1", scope), "db1", "t", () -> table("t")); + Table b = IcebergStatementScope.sharedTable(new ScopeSession(2L, "q1", scope), "db1", "t", () -> table("t")); + Assertions.assertNotSame(a, b, "different catalog id -> isolated load"); + } + + @Test + public void underNoneScopeLoadsEveryTime() { + // No live statement scope: each call loads (byte-identical to the pre-scope offline behavior). + ScopeSession none = new ScopeSession(7L, "q1", ConnectorStatementScope.NONE); + AtomicInteger loads = new AtomicInteger(); + IcebergStatementScope.sharedTable(none, "db1", "t", () -> { + loads.incrementAndGet(); + return table("t"); + }); + IcebergStatementScope.sharedTable(none, "db1", "t", () -> { + loads.incrementAndGet(); + return table("t"); + }); + Assertions.assertEquals(2, loads.get(), "NONE -> load every time"); + } + + @Test + public void rewritableDeleteSupplyIsSharedPerStatementAndIsolatedPerCatalog() { + // The scan seam and the write seam of one statement (same catalog + queryId) share ONE supply map; a + // cross-catalog MERGE keeps each catalog's supply isolated. MUTATION: dropping the catalog id from the + // key -> the two catalogs collide -> red. + TestStatementScope scope = new TestStatementScope(); + Map> supplyScan = + IcebergStatementScope.rewritableDeleteSupply(new ScopeSession(1L, "q1", scope)); + Map> supplyWrite = + IcebergStatementScope.rewritableDeleteSupply(new ScopeSession(1L, "q1", scope)); + Assertions.assertSame(supplyScan, supplyWrite, "scan and write of one statement share one supply map"); + + Map> supplyOtherCatalog = + IcebergStatementScope.rewritableDeleteSupply(new ScopeSession(2L, "q1", scope)); + Assertions.assertNotSame(supplyScan, supplyOtherCatalog, "a different catalog (cross-catalog MERGE) is isolated"); + } + + /** Minimal {@link ConnectorSession} carrying a catalog id, queryId and scope for the key + memo assertions. */ + private static final class ScopeSession implements ConnectorSession { + private final long catalogId; + private final String queryId; + private final ConnectorStatementScope scope; + + ScopeSession(long catalogId, String queryId, ConnectorStatementScope scope) { + this.catalogId = catalogId; + this.queryId = queryId; + this.scope = scope; + } + + @Override + public long getCatalogId() { + return catalogId; + } + + @Override + public String getQueryId() { + return queryId; + } + + @Override + public ConnectorStatementScope getStatementScope() { + return scope; + } + + @Override + public String getUser() { + return "u"; + } + + @Override + public String getTimeZone() { + return "UTC"; + } + + @Override + public String getLocale() { + return "en_US"; + } + + @Override + public String getCatalogName() { + return "c"; + } + + @Override + public T getProperty(String name, Class type) { + return null; + } + + @Override + public Map getCatalogProperties() { + return Collections.emptyMap(); + } + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleTest.java index 3080b461a5694b..f5e928afaaa50c 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTableHandleTest.java @@ -18,10 +18,6 @@ package org.apache.doris.connector.iceberg; import com.google.common.collect.ImmutableSet; -import org.apache.iceberg.PartitionSpec; -import org.apache.iceberg.Schema; -import org.apache.iceberg.Table; -import org.apache.iceberg.types.Types; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -29,7 +25,6 @@ import java.io.ByteArrayOutputStream; import java.io.ObjectInputStream; import java.io.ObjectOutputStream; -import java.util.Collections; /** * Tests for {@link IcebergTableHandle}, including the T07 MVCC / time-travel pin carriers and the @@ -360,58 +355,4 @@ public void topnLazyMaterializeComposesWithPinAndScope() { Assertions.assertEquals(ImmutableSet.of("oss://b/db/t1/f1.parquet"), h.getRewriteFileScope()); } - // ==================== PERF-01: query-scoped resolved-table memo ==================== - - private static Table fakeTable() { - return new FakeIcebergTable("t1", - new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get())), - PartitionSpec.unpartitioned(), "s3://b/db1/t1", Collections.emptyMap()); - } - - @Test - public void resolvedTableMemoRidesPinAndScopeCopiesButIsNotIdentity() { - Table table = fakeTable(); - IcebergTableHandle bare = new IcebergTableHandle("db1", "t1"); - Assertions.assertNull(bare.getResolvedTable(), "a fresh handle has no resolved-table memo"); - bare.setResolvedTable(table); - Assertions.assertSame(table, bare.getResolvedTable()); - - // WHY (PERF-01): getColumnHandles resolves and memoizes the table BEFORE applySnapshot copies the handle - // to thread the pin on (and rewrite scope / topn likewise). So each with* copy MUST carry the memo, else - // a pin copy would re-load the table it already resolved. MUTATION: a with* copy not carrying - // resolvedTable -> getResolvedTable() null on the copy -> red. - Assertions.assertSame(table, bare.withSnapshot(42L, null, 3L).getResolvedTable()); - Assertions.assertSame(table, bare.withRewriteFileScope(ImmutableSet.of("s3://b/db1/t1/f.parquet")) - .getResolvedTable()); - Assertions.assertSame(table, bare.withTopnLazyMaterialize(true).getResolvedTable()); - - // WHY: the memo is a resolution cache, NOT a coordinate, so it must NOT enter handle identity (the plan - // key). MUTATION: adding resolvedTable to equals/hashCode -> a resolved handle would not equal an - // un-resolved handle for the same table -> the two asserts below fail. - IcebergTableHandle unresolved = new IcebergTableHandle("db1", "t1"); - Assertions.assertEquals(unresolved, bare, "the memo must not change equals"); - Assertions.assertEquals(unresolved.hashCode(), bare.hashCode(), "the memo must not change hashCode"); - } - - @Test - public void resolvedTableMemoIsTransientAcrossSerialization() throws Exception { - IcebergTableHandle original = new IcebergTableHandle("db1", "t1"); - original.setResolvedTable(fakeTable()); - - // The handle serializes even though the memoized FakeIcebergTable is NOT Serializable — proving the - // field is transient. MUTATION: making resolvedTable non-transient -> writeObject throws - // NotSerializableException here -> red. - ByteArrayOutputStream baos = new ByteArrayOutputStream(); - try (ObjectOutputStream oos = new ObjectOutputStream(baos)) { - oos.writeObject(original); - } - IcebergTableHandle restored; - try (ObjectInputStream ois = new ObjectInputStream(new ByteArrayInputStream(baos.toByteArray()))) { - restored = (IcebergTableHandle) ois.readObject(); - } - // WHY: a deserialized handle (BE side / plan reuse) must start with NO memo and resolve live — the live - // SDK Table is FE-only. MUTATION: a non-transient memo would either fail serialization or staple a stale - // table onto the wire object. - Assertions.assertNull(restored.getResolvedTable(), "the resolved-table memo must not cross serialization"); - } } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java index 15126a6a2d163b..caf52891f480b4 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergWritePlanProviderTest.java @@ -19,6 +19,7 @@ import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; @@ -803,18 +804,17 @@ public void planWriteDeleteSinkFormatVersionThree() { "a v3 delete with no live delete files to rewrite must not set rewritable sets (legacy gates on non-empty)"); } - // ── commit-bridge supply (S4 part 2): planWrite drains the scan-time stash into rewritable_delete_file_sets ── + // ── commit-bridge supply (S4 part 2): planWrite drains the per-statement scope into rewritable_delete_file_sets ── private static final String STASH_QID = "qid-stash"; - private static IcebergWritePlanProvider providerWithStash(Table table, RecordingConnectorContext ctx, - IcebergRewritableDeleteStash stash) { + private static IcebergWritePlanProvider supplyProvider(Table table, RecordingConnectorContext ctx) { RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); ops.table = table; - return new IcebergWritePlanProvider(NON_REST_PROPS, ops, ctx, stash); + return new IcebergWritePlanProvider(NON_REST_PROPS, ops, ctx); } - private static WriteSession stashSession(Table table, RecordingConnectorContext ctx, String queryId) { + private static WriteSession supplySession(Table table, RecordingConnectorContext ctx, String queryId) { RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); ops.table = table; IcebergConnectorTransaction txn = new IcebergConnectorTransaction(42L, ops, ctx); @@ -831,18 +831,18 @@ private static List dvDescs(String path, long offset, lo } @Test - public void planWriteDeleteSinkAttachesRewritableSetsFromStashAndEvicts() { - // The scan stashed this statement's old DV (referenced data file -> its non-equality deletes); planWrite - // must drain it onto the sink so the BE OR-merges those deletes into the new DV. MUTATION: not reading - // the stash / not setting the field -> the BE writes a DV without the old deletes -> resurrection. + public void planWriteDeleteSinkAttachesRewritableSetsFromScope() { + // The scan accumulated this statement's old DV (referenced data file -> its non-equality deletes) into the + // per-statement scope; planWrite must drain it onto the sink so the BE OR-merges those deletes into the + // new DV. MUTATION: not reading the scope / not setting the field -> the BE writes a DV without the old + // deletes -> resurrection. Table table = formatVersionThreeTable(freshCatalog()); RecordingConnectorContext ctx = contextWithStorage(); - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate(STASH_QID, "oss://bucket/wh/db1/tv3/data/f1.parquet", + WriteSession session = supplySession(table, ctx, STASH_QID); + IcebergStatementScope.rewritableDeleteSupply(session).put("oss://bucket/wh/db1/tv3/data/f1.parquet", dvDescs("oss://bucket/wh/db1/tv3/data/dv1.puffin", 16L, 64L)); - ConnectorSinkPlan plan = providerWithStash(table, ctx, stash).planWrite( - stashSession(table, ctx, STASH_QID), + ConnectorSinkPlan plan = supplyProvider(table, ctx).planWrite(session, new WriteHandle(new IcebergTableHandle("db1", "tv3")).writeOperation(WriteOperation.DELETE)); TIcebergDeleteSink sink = plan.getDataSink().getIcebergDeleteSink(); @@ -854,20 +854,17 @@ public void planWriteDeleteSinkAttachesRewritableSetsFromStashAndEvicts() { Assertions.assertEquals(1, set.getDeleteFilesSize()); Assertions.assertEquals("oss://bucket/wh/db1/tv3/data/dv1.puffin", set.getDeleteFiles().get(0).getPath()); Assertions.assertEquals(3, set.getDeleteFiles().get(0).getContent()); - // The retrieve is the primary eviction; a second statement must not re-read this entry. - Assertions.assertEquals(0, stash.size(), "planWrite must evict the stash entry it consumed"); } @Test - public void planWriteMergeSinkAttachesRewritableSetsFromStash() { + public void planWriteMergeSinkAttachesRewritableSetsFromScope() { Table table = formatVersionThreeTable(freshCatalog()); RecordingConnectorContext ctx = contextWithStorage(); - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate(STASH_QID, "oss://bucket/wh/db1/tv3/data/f1.parquet", + WriteSession session = supplySession(table, ctx, STASH_QID); + IcebergStatementScope.rewritableDeleteSupply(session).put("oss://bucket/wh/db1/tv3/data/f1.parquet", dvDescs("oss://bucket/wh/db1/tv3/data/dv1.puffin", 8L, 32L)); - ConnectorSinkPlan plan = providerWithStash(table, ctx, stash).planWrite( - stashSession(table, ctx, STASH_QID), + ConnectorSinkPlan plan = supplyProvider(table, ctx).planWrite(session, new WriteHandle(new IcebergTableHandle("db1", "tv3")).writeOperation(WriteOperation.MERGE)); TIcebergMergeSink sink = plan.getDataSink().getIcebergMergeSink(); @@ -875,63 +872,72 @@ public void planWriteMergeSinkAttachesRewritableSetsFromStash() { "a v3 MERGE must carry rewritable_delete_file_sets (thrift field 25) too"); Assertions.assertEquals("oss://bucket/wh/db1/tv3/data/f1.parquet", sink.getRewritableDeleteFileSets().get(0).getReferencedDataFilePath()); - Assertions.assertEquals(0, stash.size()); } @Test - public void planWriteDeleteSinkLeavesSetsUnsetWhenNoStashEntryForQuery() { - // A v3 DELETE whose scan stashed nothing for this queryId (no live deletes) leaves the field unset — - // byte-identical to legacy's empty gate. + public void planWriteDeleteSinkLeavesSetsUnsetWhenScopeHasNoSupply() { + // A v3 DELETE whose scan accumulated nothing into the scope (no live deletes) leaves the field unset — + // byte-identical to legacy's empty gate. The queryId scoping means a different statement's supply is + // simply not visible here. Table table = formatVersionThreeTable(freshCatalog()); RecordingConnectorContext ctx = contextWithStorage(); - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate("some-other-query", "oss://bucket/wh/db1/tv3/data/f1.parquet", - dvDescs("dv", 1L, 2L)); - ConnectorSinkPlan plan = providerWithStash(table, ctx, stash).planWrite( - stashSession(table, ctx, STASH_QID), + ConnectorSinkPlan plan = supplyProvider(table, ctx).planWrite( + supplySession(table, ctx, STASH_QID), new WriteHandle(new IcebergTableHandle("db1", "tv3")).writeOperation(WriteOperation.DELETE)); Assertions.assertFalse(plan.getDataSink().getIcebergDeleteSink().isSetRewritableDeleteFileSets()); - // The other query's entry is untouched. - Assertions.assertEquals(1, stash.size()); } @Test - public void planWriteVersionTwoDeleteNeverAttachesRewritableSetsButStillEvicts() { - // v2 deletes are plain position-delete files (no DV union), so even a stashed supply must NOT be emitted - // (the BE ignores field 15 below v3). MUTATION: dropping the formatVersion>=3 gate would emit it. The - // retrieve still evicts (no leak). + public void planWriteVersionTwoDeleteNeverAttachesRewritableSets() { + // v2 deletes are plain position-delete files (no DV union), so even a scope-carried supply must NOT be + // emitted (the BE ignores field 15 below v3). MUTATION: dropping the formatVersion>=3 gate would emit it. InMemoryCatalog catalog = freshCatalog(); Table table = catalog.createTable(TableIdentifier.of("db1", "tv2"), SCHEMA, PartitionSpec.unpartitioned(), Collections.singletonMap("format-version", "2")); RecordingConnectorContext ctx = contextWithStorage(); - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate(STASH_QID, "oss://bucket/wh/db1/tv2/data/f1.parquet", dvDescs("dv", 1L, 2L)); + WriteSession session = supplySession(table, ctx, STASH_QID); + IcebergStatementScope.rewritableDeleteSupply(session) + .put("oss://bucket/wh/db1/tv2/data/f1.parquet", dvDescs("dv", 1L, 2L)); - ConnectorSinkPlan plan = providerWithStash(table, ctx, stash).planWrite( - stashSession(table, ctx, STASH_QID), + ConnectorSinkPlan plan = supplyProvider(table, ctx).planWrite(session, new WriteHandle(new IcebergTableHandle("db1", "tv2")).writeOperation(WriteOperation.DELETE)); Assertions.assertFalse(plan.getDataSink().getIcebergDeleteSink().isSetRewritableDeleteFileSets()); - Assertions.assertEquals(0, stash.size(), "the retrieve evicts even when the v3 gate drops the supply"); } @Test - public void planWriteInsertEvictsStashEntryForTheStatement() { - // INSERT ... SELECT FROM an iceberg source stashes the source scan's deletes, but the INSERT write does - // not consume them; planWrite must still evict the entry so it does not leak. MUTATION: gating the - // retrieve on DELETE/MERGE only would leak the INSERT path's entry. + public void planWriteInsertIgnoresScopeSupply() { + // INSERT ... SELECT FROM an iceberg source accumulates the source scan's deletes into the scope, but the + // INSERT write does not consume them; planWrite must still produce a plain table sink (the supply is + // simply ignored, not attached). MUTATION: attaching the supply to an INSERT sink -> red. Table table = formatVersionThreeTable(freshCatalog()); RecordingConnectorContext ctx = contextWithStorage(); - IcebergRewritableDeleteStash stash = new IcebergRewritableDeleteStash(); - stash.accumulate(STASH_QID, "oss://bucket/wh/db1/tv3/data/src.parquet", dvDescs("dv", 1L, 2L)); + WriteSession session = supplySession(table, ctx, STASH_QID); + IcebergStatementScope.rewritableDeleteSupply(session) + .put("oss://bucket/wh/db1/tv3/data/src.parquet", dvDescs("dv", 1L, 2L)); - providerWithStash(table, ctx, stash).planWrite( - stashSession(table, ctx, STASH_QID), + ConnectorSinkPlan plan = supplyProvider(table, ctx).planWrite(session, new WriteHandle(new IcebergTableHandle("db1", "tv3")).writeOperation(WriteOperation.INSERT)); - Assertions.assertEquals(0, stash.size(), "an INSERT write still evicts its statement's stash entry"); + Assertions.assertTrue(plan.getDataSink().isSetIcebergTableSink(), "INSERT builds a plain table sink"); + } + + @Test + public void planWriteV3RowLevelDmlUnderNoneScopeFailsLoud() { + // A format-version>=3 DELETE/MERGE under NONE (no live statement scope) cannot have received the scan's + // rewritable-delete supply, so planWrite must reject it rather than silently write a DV that drops old + // deletes. MUTATION: dropping the fail-loud -> a v3 DELETE under NONE resurrects previously-deleted rows. + Table table = formatVersionThreeTable(freshCatalog()); + RecordingConnectorContext ctx = contextWithStorage(); + WriteSession session = supplySession(table, ctx, STASH_QID).noScope(); + + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, () -> + supplyProvider(table, ctx).planWrite(session, + new WriteHandle(new IcebergTableHandle("db1", "tv3")).writeOperation(WriteOperation.DELETE))); + Assertions.assertTrue(ex.getMessage().contains("per-statement scope"), + "fail-loud message must name the missing per-statement scope, got: " + ex.getMessage()); } @Test @@ -1159,6 +1165,9 @@ public TSortInfo getSortInfo() { private static final class WriteSession implements ConnectorSession { private final ConnectorTransaction txn; private final String queryId; + // Default to a live memoizing scope so a v3 row-level DML does not trip planWrite's NONE fail-loud; the + // dedicated fail-loud test forces NONE via noScope(). + private ConnectorStatementScope statementScope = new TestStatementScope(); WriteSession(ConnectorTransaction txn) { this(txn, "q"); @@ -1169,6 +1178,23 @@ private static final class WriteSession implements ConnectorSession { this.queryId = queryId; } + /** Shares an explicit scope (so a test can pre-fill the rewritable-delete supply the write then reads). */ + WriteSession withScope(ConnectorStatementScope scope) { + this.statementScope = scope; + return this; + } + + /** Forces ConnectorStatementScope.NONE (for the v3-DML-under-NONE fail-loud test). */ + WriteSession noScope() { + this.statementScope = ConnectorStatementScope.NONE; + return this; + } + + @Override + public ConnectorStatementScope getStatementScope() { + return statementScope; + } + @Override public Optional getCurrentTransaction() { return Optional.ofNullable(txn); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TestStatementScope.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TestStatementScope.java new file mode 100644 index 00000000000000..e6744a21e0948a --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TestStatementScope.java @@ -0,0 +1,41 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.api.ConnectorStatementScope; + +import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Supplier; + +/** + * A memoizing {@link ConnectorStatementScope} for iceberg unit tests. The iceberg connector module does not + * depend on fe-core, so tests cannot use the engine's {@code ConnectorStatementScopeImpl}; this is a faithful + * copy of it. Sharing one instance across a scan session and a write session mimics a single statement's scope, + * so a test can prove the read/scan/write resolvers collapse onto one load and that the rewritable-delete supply + * bridges scan→write. + */ +final class TestStatementScope implements ConnectorStatementScope { + + private final ConcurrentHashMap cache = new ConcurrentHashMap<>(); + + @Override + @SuppressWarnings("unchecked") + public T computeIfAbsent(String key, Supplier loader) { + return (T) cache.computeIfAbsent(key, k -> loader.get()); + } +} From 19ac5da284cd0f8636c20a5eafdf32f99abe4a15 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 20:51:11 +0800 Subject: [PATCH 18/47] [doc](catalog) fe-connector-iceberg PERF-07: design summary + tracker updates; next = PERF-08 Record the completed PERF-07 (unified per-statement table-load owner): summary doc, progress entry (6-way recon + adversarial review notes + build gotchas), tasklist row flipped to done, and a fresh HANDOFF pointing at PERF-08 (the rewrite_data_files / expire_snapshots maintenance-path re-scan/dedup fixes). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 63 +++++++++---------- ...ified-per-statement-table-owner-summary.md | 49 +++++++++++++++ plan-doc/perf-hotpath-iceberg/progress.md | 17 +++++ plan-doc/perf-hotpath-iceberg/tasklist.md | 12 ++-- 4 files changed, 104 insertions(+), 37 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-summary.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index f446c19a0e066e..76a223189fffdc 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -6,57 +6,54 @@ --- -# ✅ 已完成（PERF-01 ~ 06） +# ✅ 已完成（PERF-01 ~ 07） - **PERF-01**（`484f0e0c125`）：胖 handle + 跨查询 `IcebergTableCache`（**凭证 gate**）。规划期 loadTable 3~7→1。 -- **PERF-02**（`518d0599cbf`）：`IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、无 gate）。分区扫描每查询→每快照。 -- **PERF-03**（`0b96f2e6c78`）：`IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、无 gate）。格式推断每查询→每快照。 -- **PERF-04**（`2e5f393779c`）：抽惰性 `cacheBackedFileScanTasks` 三处复用，manifest cache 接回大表流式 + COUNT(*)（缓存与不 OOM 兼得）。 -- **PERF-05**（`aea3ebdd40e`）：`IcebergCommentCache`（键 `TableIdentifier`、无 gate），**仅 vended 非 session=user**。information_schema comment 每表 load 跨查询坍缩。 -- **PERF-06**（`6294edf2833`）：vended-credentials URI 归一化 **scan 级「准备一次、逐文件套用」**。新增 SPI `newStorageUriNormalizer`（默认逐文件折回、零影响其它连接器），`DefaultConnectorContext` override 惰性 memo 一次派生；iceberg 三 scan-start 各构造一次 normalizer 穿六 seam；`TcclPinning` 透传。**否决 fe-core 框架 memo**（per-catalog 跨查询共享→并发冲刷+凭证保留）。一次 vended scan `StorageProperties.createAll` O(N_files+N_deletes)→1。小结 `designs/FIX-PERF-06-*-summary.md`。 -- iceberg 全模块单测绿（→974 / 0 fail / 1 skip），fe-core `DefaultConnectorContextNormalizeUriTest` 13 绿，checkstyle 绿，0 回归。 -- **可复用产物**：四套缓存基建（table/partition/format/comment）+ 一个 SPI「准备一次、套用多次」normalizer 范式；**gate 判据三条**（①值含凭证→gate ②纯元数据→无 gate ③授权在 load 里/session=user→不可共享）；**PERF-06 新判据「作用域即安全边界」**：同一份含凭证派生，跨查询做撞 gate、**scan 级做天然安全**（token 恒定、无跨用户、结束即回收）。 +- **PERF-02**（`518d0599cbf`）：`IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、无 gate）。 +- **PERF-03**（`0b96f2e6c78`）：`IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、无 gate）。 +- **PERF-04**（`2e5f393779c`）：抽惰性 `cacheBackedFileScanTasks` 三处复用，manifest cache 接回大表流式 + COUNT(*)。 +- **PERF-05**（`aea3ebdd40e`）：`IcebergCommentCache`（键 `TableIdentifier`、无 gate），**仅 vended 非 session=user**。 +- **PERF-06**（`6294edf2833`）：vended-credentials URI 归一化 scan 级「准备一次、逐文件套用」normalizer。 +- **PERF-07**（`97bdcd6bdbe` fe-core + `ea7fd1f6e7a` iceberg）：每语句 `ConnectorStatementScope`（挂 `StatementContext`、构造期捕获）作读写共享的表加载归属者，一条语句一张表只加载一次；拆胖句柄；delete stash 下沉到作用域（删单例 + `IcebergRewritableDeleteStash` 类 + 测试）；v3 行级 DML + NONE fail-loud。写侧收益≈0，交付=架构连贯。**净增 fe-core SPI（碰铁律 A，用户签字）**。设计+小结见 `designs/FIX-PERF-07-unified-*`。 +- iceberg 全模块单测绿（**968 pass / 1 skip**），fe-core 两段验 `ConnectorStatementScopeTest`+`ConnectorSessionImplTest` 绿，0 回归。 --- -# 🆕 下一个 session = **PERF-07（C20 写路径一条 DML 3~5 次 load 同表）** +# 🆕 下一个 session = **PERF-08（维护路径逐单位重扫 / 无去重，C19+C21）** -## 病灶（审计基线行号，动码前必重 grep） +> 改动小、收益明确、**不碰读写热路径**（相较 PERF-07 是轻活）；覆盖两条独立维护命令，同一「逐单位重扫、零去重」模式；实现上可各自独立 commit。**行号信 grep**（PERF-07 后可能又漂）。 -一条 iceberg DML（INSERT/DELETE/MERGE）在规划+绑定期对同一张表串行 load 3~5 次： -`PhysicalIcebergMergeSink.getRequirePhysicalProperties:161→198`、`PhysicalPlanTranslator.visitPhysicalConnectorTableSink:675,703`、`PluginDrivenTableSink.bindDataSink:175` → `IcebergWritePlanProvider.resolveTable:689-702` / `beginWrite`（内含 `tableExists` ×2 + **无条件 refresh**）。每条 DML +3~5 次串行远程往返。 -**修复方向（审计）**：语句级 resolve 一次传递；`exists` 从 load 结果推导（load 成功即存在，不必单独 tableExists）。 +## C19 — `rewrite_data_files`（分布式 compaction 规划） +- **病灶**：`ConnectorRewriteDriver.run` STEP3（约 :143）**每个** rewrite group 各调一次 `registerRewriteSourceFiles` → 每次一遍 `useSnapshot(...).planFiles()`（整表 manifest 扫描）。G 个 group = **G+1 次整表扫描**，G~50-200 时分钟级。 +- **修复方向**：**union 所有 group 一次注册**（SPI `registerRewriteSourceFiles` 本就收 `Set`）——把 G 次调用坍缩为 1 次，扫一遍 planFiles、按 group 分派。 +- **⚠ 落地前 grep 定位**：`ConnectorRewriteDriver`（fe-core 通用驱动？还是连接器侧？）的 STEP3 循环真实结构 + `registerRewriteSourceFiles` 的连接器实现（iceberg 侧 `IcebergConnectorTransaction.registerRewriteSourceFiles`，PERF-07 recon 记其在 `:362`、re-derive 在 `:380-381`、当前 `startingSnapshotId`）；确认「union 一次注册」不破分布式 rewrite 的 per-group 文件归属/OCC。 -## 第一件事（立项流程见 README §单项立项流程） +## C21 — `expire_snapshots` +- **病灶**：`IcebergExpireSnapshotsAction.buildDeleteFileContentMap`（约 :271-293）对**每个** snapshot 读其全部 delete manifest、**无 visited-path 去重** —— 相邻 snapshot 的 manifest 大量重叠，S×M 串行远程读。 +- **修复方向**：按 `ManifestFile.path()` **去重坍缩为 O(distinct)**（visited set）。 +- **⚠ 落地前 grep 定位**：`IcebergExpireSnapshotsAction` 真实结构 + 该 map 的消费方（确认按 path 去重不丢任何该删的 content 条目）。 -1. **复核（动码前，行号信 grep；PERF-01~06 后已漂移）**：grep 上述五处 load/resolve/beginWrite/tableExists 调用点，重新确认：①这些 load 是否已被 PERF-01 的 `IcebergTableCache` 部分兜住（写路径是否走 resolveTableForRead？还是另一条 write-only 的 resolve/refresh？**PERF-05 复核先例：审计早于 PERF-01，残余可能已缩小**）②`beginWrite` 的**无条件 refresh** 是真必需还是可省（refresh 破坏 PERF-01 缓存命中？）③`tableExists ×2` 能否从 load 结果推导。 -2. **⚠ 路由/归属**：写路径 resolve 分散在 **Nereids 物理算子层（fe-core `PhysicalPlanTranslator`/`PluginDrivenTableSink`）** 与 **连接器 `IcebergWritePlanProvider`** 两侧。语句级"resolve 一次传递"若要跨这两层，可能须动 fe-core 的 sink 绑定流程——**先核能否纯连接器侧收敛**（`IcebergWritePlanProvider` 内 statement 级 memo，键=queryId/statementId + TableIdentifier），若须动 fe-core 通用 sink 流程则**动码前上交用户**（对齐 PERF-06/铁律）。 -3. **红队 + TDD**：度量守门 = 一条 DML 内对同表的 loadTable 远端次数从 3~5→1（可加计数器/spy，镜像 PERF-01 的 `loadCountForTest`）。**行为 parity 是硬闸门**：写语义（exists 判定、refresh 拿到最新快照）不得变——若省 refresh，须证写入仍绑定到正确的最新 metadata。 -4. 守铁律：优先连接器侧；exists 从 load 推导别新增 RPC；别把 Table 缓存塞进 fe-core。 - -## ⚠️ 关键认知 - -- **写路径 refresh 与 PERF-01 读缓存的张力**：PERF-01 的 `IcebergTableCache` 按 `(TableIdentifier, snapshotId)` 缓存；写路径若无条件 `refresh()` 会绕过/失效它。核清"写必须看最新"与"缓存命中"能否兼得（可能写路径本就该 pin 一次最新快照后全语句复用，类似 PERF-01/02 的 pin 模式）。 -- **statement 级作用域**：与 PERF-06 的 scan 级同理——写路径的 load 复用作用域是**一条语句**（queryId/statementId 稳定），非跨查询，故不涉凭证/授权 gate；memo 挂 statement context 或连接器侧按 queryId 键。 -- **exists 从 load 推导**：`tableExists` 单独 RPC 多半可删——load 成功即存在、load 抛 NoSuchTable 即不存在；但要核**建表/CTAS 分支**（表尚不存在时 exists=false 是正常路径，不能把"不存在"当错误抛）。 +## 铁律提醒（PERF-08） +- 维护路径修复**不碰** fe-core 读写热缝、不碰 PERF-07 的作用域；若发现 C19 的驱动在 fe-core 通用层，遵「共享框架须 byte+cost 双不变」纪律 + 保持 connector-agnostic（对齐 `catalog-spi-plugindriven-no-source-specific-code`）。 +- 立项先读审计报告 §5 对 C19/C21 的原始证据，再对照 HEAD 真实代码 review（HANDOFF 行号/方案可能过时）。 --- # 🧰 构建/验证坑（**实证，务必照做**） -1. **iceberg 侧测试**：`mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`（绝对 `-f /fe/pom.xml`）。原因见旧注：`${revision}` 未 flatten → 必须 `-am`；`-am test` 不产 shade jar 故用 **`install`**；`-Dtest=<类列表>`（`ls src/test/.../*Test.java` 拼逗号，本轮 56 类，存 `scratchpad/iceberg-tests.txt`）让上游快跳。 -2. **⚠ 改到 fe-core 的项须单独验 fe-core**（PERF-06 实证）：**iceberg 连接器不依赖 fe-core**（SPI 解耦），`-pl iceberg -am` 反应堆里**没有 fe-core**——fe-core 的改动 + 其单测须**另跑** `mvn test -pl fe-core -am -Dtest= -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -f /fe/pom.xml`。PERF-07 若动 fe-core sink 流程，两段都要验。 +1. **iceberg 侧测试**：`mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='<类逗号列表>' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`（绝对 `-f /fe/pom.xml`）。`${revision}` 未 flatten→必 `-am`；`-am test` 不产 shade jar 故用 **`install`**；类列表存 `scratchpad/iceberg-tests.txt`（`ls src/test/.../*Test.java | xargs -n1 basename | sed 's/.java//' | paste -sd,`）。 +2. **动 fe-core 者必两段验**：iceberg 连接器不依赖 fe-core，`-pl iceberg -am` 反应堆无 fe-core——fe-core 改动 + 其单测须另跑 `mvn test -pl fe-core -am -Dtest= -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -f /fe/pom.xml`。（PERF-08 若纯 iceberg 则免；若碰 fe-core 驱动则要。） 3. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 静默跳过）。 -4. **别用 `mvn -q` 跑测试**：成功时抑制 `BUILD SUCCESS`/`Tests run` INFO 行。用 surefire XML 或 grep `Tests run:` 聚合验证。 -5. **别 `nohup ... &` 套 `run_in_background`**——maven 变孤儿、"exit 0" 假信号。直接 `mvn ... >> log 2>&1`（run_in_background）；**通知里的 exit code 是 echo 的**，要 grep 日志 `BUILD SUCCESS`。 -6. **checkstyle 主源 ≤120**（test 源 suppress）；import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行；新用类记得 import（本轮四文件各加 `java.util.function.UnaryOperator`）。 -7. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。 +4. **别用 `mvn -q` 跑测试**：抑制 `BUILD SUCCESS`/`Tests run`。用 surefire XML 或 grep `Tests run:`。 +5. **别 `nohup ... &` 套 `run_in_background`**——maven 变孤儿、假 exit 0。直接 `mvn ... >> log 2>&1`（run_in_background）；**通知里 exit code 是 echo 的**，grep 日志 `BUILD SUCCESS`。 +6. **checkstyle 主源 ≤120**，且**扫 test 源**（PERF-07 踩坑：删测试后遗留 unused import 挂 checkstyle）——删测试后 grep 残留 import。import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行。 +7. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add（`git rm` 会自动 stage 删除，拆 commit 时用 `git restore --staged` 剔出）。 8. **并发探测**：本目录 `find fe -newermt '-120 sec'` + `pgrep -a -f maven`。`/mnt/disk1/yy/git/doris` 是另一 worktree，不冲突。 9. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 --- -# 🗂 开放问题 +# 🗂 PERF-07 遗留（已闭环，仅记录） -- **PERF-07 路由（纯连接器 statement-memo vs 动 fe-core sink 绑定流程）**：写路径 resolve 横跨 Nereids 物理算子（fe-core）与 `IcebergWritePlanProvider`（连接器）。先核能否在 `IcebergWritePlanProvider` 内按 queryId/statementId 收敛（连接器侧）；若"resolve 一次传递"须跨 fe-core sink 层，则**动码前把方案上交用户定**（对齐 PERF-06 先例）。 -- **refresh 必要性**：`beginWrite` 无条件 refresh 是否可省/可收敛为语句级 pin 一次，须与 PERF-01 读缓存的命中兼得 + 证写语义 parity。 +- e2e 回归（独立 + HMS 网关目录 INSERT/DELETE/MERGE/OVERWRITE 无复活 + 分布式 rewrite + 跨 catalog MERGE + session=user/vended 归一）**留 P6.6 切换阶段统一补**（对齐 `hms-iceberg-delegation-needs-e2e`；当前 iceberg 未在 `SPI_READY_TYPES`，provider 休眠）。 +- 架构记忆 `iceberg-table-resolution-cache-scoping` 待按本次落地更新（原记「stash 下单例待用户定高度」已由完整统一版闭环）。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-summary.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-summary.md new file mode 100644 index 00000000000000..079c55202a2f9c --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-summary.md @@ -0,0 +1,49 @@ +# FIX-PERF-07 小结 — 每语句「表加载归属者」，读写共享一次加载 + +> 覆盖审计发现 **C20**（P1）。用户 2026-07-18 拍板走「完整统一版」（读/写共享同一对象、拆胖句柄、把删除清单暂存下沉），明确**不追性能、接受净增 fe-core SPI**，目标=架构连贯。 +> 权威设计：[`FIX-PERF-07-unified-per-statement-table-owner-design.md`](./FIX-PERF-07-unified-per-statement-table-owner-design.md)。 + +## Problem + +一条 DML（尤其 DELETE/MERGE）里同一张 iceberg 表被反复远端 `loadTable`：读元数据、扫描规划、写成形（算排序列/分区/EXPLAIN 各一次）、`beginWrite` 各自加载，最糟一条语句 3~5 次。原有两层缓存只覆盖读扫描一段：胖句柄（`IcebergTableHandle.resolvedTable`，仅同一 handle 实例）+ 跨查询 `IcebergTableCache`（对 `session=user`/REST vended 关闭）；写臂完全绕过。 + +## Root Cause + +连接器 session 一条语句内被重建约 26 次，缓存挂它即死；三个表解析器（读/扫描/写）各自为政、写事务又新载。唯一贯穿整条语句的对象是 fe-core 的 `StatementContext`——连接器需向上够到它。对齐 Trino「每事务一个 metadata + tableMetadataCache」：Doris 无「每事务 metadata」，可移植的落点=把每语句作用域挂 `StatementContext`，经 `ConnectorSession.getStatementScope()` 够到。 + +## Fix + +**中性 SPI（`fe-connector-api`）** +- 新增 `ConnectorStatementScope`：` T computeIfAbsent(String key, Supplier)` + `NONE`（不缓存=逐次加载，离线/无上下文默认）。 +- `ConnectorSession` 加默认方法 `getStatementScope()`→`NONE`（其余 20 个实现全测试用、零改，自动继承 NONE）。 + +**fe-core** +- `ConnectorStatementScopeImpl`（`ConcurrentHashMap` 背书；连接器 loader 不回环，`computeIfAbsent` 单键原子安全）。 +- `StatementContext`：懒建字段 + `synchronized` 访问器（镜像 `getOrCacheDisableRules`），**不在 `close()`/`releasePlannerResources` 清**（随 GC，镜像 `snapshots`）；`resetConnectorStatementScope()`。 +- `ConnectorSessionImpl`/`Builder`：**构造期捕获**作用域引用（`from(ctx)` 的 ctx 优先，回退 `ConnectContext.get()`，两级 null→NONE，镜像 `MvccUtil.getSnapshotFromContext`）——因扫描流式/分批在 off-thread 池跑、复用请求线程建的**同一** session，实时读 thread-local 会失效。 +- `ExecuteCommand`：预编译 EXECUTE 复用同一 `StatementContext`，在既有每执行重置旁加一行 `resetConnectorStatementScope()`（唯一复用路径、每执行恰一次、规划前；作用域 key 含 queryId 为第二道防线）。 + +**iceberg** +- 新增连接器私有 `IcebergStatementScope`：`sharedTable(session, db, tbl, loader)`（键 `iceberg.table:catalogId:db:tbl:queryId`）+ `rewritableDeleteSupply(session)`（键 `iceberg.rewritable-delete-supply:catalogId:queryId`）；`session==null` 时逐次加载/抛式 map（等价 NONE）。 +- 读 `resolveTableForRead`、扫描 `resolveTable`、写 `resolveTable`、`beginWrite` 四处全改走 `sharedTable`；loader 各自保留原授权（读 `catalogOps` / 扫描写 `catalogOpsResolver.apply(session)`，均 `newCatalogBackedOps(session)`，同用户）+ L1 层（扫描/读）。 +- **拆 L0**：删 `IcebergTableHandle.resolvedTable` 字段/访问器/三处 `with*` 携带 + 两 seam 读写；句柄回归纯坐标。 +- `beginWrite` 取共享表（DELETE/MERGE 走扫描已填的作用域=命中，写-only INSERT 才加载一次）；`openTransaction` **保持** `newTransaction()` 的 refresh（给提交新鲜 OCC 基底；读扫描显式钉快照读不可变数据，共享对象被 refresh 到 latest 无害且顺带消解跨缓存错位）。 +- **删除清单下沉**：扫描 `buildRangeForTask` accumulate → 作用域同键 map.put；写 `planWrite` 读同键 map；**整删** `IcebergRewritableDeleteStash`（141 行）+ 单例字段 + 两 provider 四/二个 ctor 的 stash 参数 + 其测试（204 行）。 +- **响亮失败**：`formatVersion>=3 && (DELETE|UPDATE|MERGE) && getStatementScope()==NONE` → 抛（迁移唯一新增的「静默复活」风险降为响亮失败；生产恒有 StatementContext，仅离线触发）。 + +## 关键取舍（诚实） + +- **写侧性能收益 ≈ 0**：DELETE/MERGE 写解析命中作用域（扫描已载）、INSERT 仍一次；`beginWrite` 保留 refresh。真交付=**架构连贯 + 删单例/删类**（原则性，非修 bug）。 +- **`beginWrite` 保留 refresh（非 drop）**：设计原倾向 drop 以最大连贯，但 `Transactions.newTransaction(name, ops)` 本身会 `ops.refresh()`；保留它更安全（新鲜 OCC 基底 + 消解「共享表比 pin 旧」的跨缓存错位），且共享对象被 refresh 到 latest 对已显式钉快照的读扫描无害。 +- **跨缓存快照错位不新增**：pin 由 `IcebergLatestSnapshotCache` 铸、共享表由 `IcebergTableCache`/直载——该错位今天已存在于读扫描路径；写扫描共享读已用过的同一对象，故只在读本就会抛处抛，严格不更差。 +- **授权**：读/扫描/写均 `newCatalogBackedOps(session)`，一语句=一用户=一凭证；作用域存 RAW、session 生死、key 含 catalogId+queryId；`session=user`/vended 全开（L1 对其关，作用域可开，凭证目录受益最大）。 + +## 测试与守门 + +- **iceberg（968 pass / 0 fail / 1 skip）**：`planningPassLoadsSameTableOnceViaSharedScope`（读+扫描共享作用域→远端 `loadTable` 计数 1）+ 对照 `...WithoutSharedScopeLoadsEachTime`（NONE→2）；扫描 accumulate / 写 drain 经作用域（v3 命中、v2 不填、NONE inert）；写 fail-loud（v3 DELETE + NONE → 抛，消息含 "per-statement scope"）；`IcebergStatementScopeTest`（同键共享 = 读写同实例 / 跨 queryId 隔离 = 预编译重执行 / 跨 catalogId 隔离 = 跨-catalog MERGE / NONE 逐次 / 供给图同语句共享跨 catalog 隔离）。删 `IcebergTableHandleTest` 两个 L0 测试 + `IcebergRewritableDeleteStashTest`。 +- **fe-core（两段验，iceberg 反应堆不含 fe-core）**：`ConnectorStatementScopeTest`（NONE 不 memo / impl 每键 memo+隔离 / `StatementContext` 懒建稳定 + reset 使下次全新、旧值不泄漏）3 绿；`ConnectorSessionImplTest` 17 绿（ctor 新增作用域参数无回退）。 +- **e2e（后续切换阶段补，对齐 `hms-iceberg-delegation-needs-e2e`）**：独立 + HMS 网关目录 INSERT/DELETE/MERGE/OVERWRITE 无复活 + 分布式 rewrite + 跨 catalog MERGE；session=user/vended 归一。 + +## 净增 fe-core（碰铁律 A，用户签字） + +`fe-connector-api`：`ConnectorStatementScope` + `getStatementScope()` 默认方法。`fe-core`：`ConnectorStatementScopeImpl` + `StatementContext` 字段/访问器/重置 + `ConnectorSessionImpl`/`Builder` 构造期捕获 + `ExecuteCommand` 一行。连接器仍不解析属性、作用域值为 `Object`（fe-core 不知 iceberg 类型，connector-agnostic）。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index f6427f96d5f05c..d4b9747f8a5634 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -99,3 +99,20 @@ - **结果**：iceberg 全模块 **974 pass / 0 fail / 1 skip**（+1 守门）；fe-core `DefaultConnectorContextNormalizeUriTest` **9→13 绿**；两处 BUILD SUCCESS + checkstyle 0 违规，0 回归。summary 见 `designs/FIX-PERF-06-*-summary.md`。 - **新判据（可复用）**：**「作用域即安全边界」**——同一份"含凭证派生"的缓存，跨查询（catalog 级对象）做=撞凭证/授权 gate；**scan 级**做=天然安全（token 恒定、无跨用户、结束即回收）。PERF-05 的 gate 是"跨查询缓存必须判授权"，PERF-06 是"把作用域收回 scan 就不必判"——两面。 - **下一步**：见 HANDOFF —— PERF-07（C20 写路径一条 DML 3~5 次 load 同表）。 + +--- + +## 2026-07-18 — session 5：PERF-07 实现（完整统一版）+ 全绿（commit `97bdcd6bdbe` fe-core + `ea7fd1f6e7a` iceberg） + +- **开场校验（6 路侦察 workflow）**：把权威设计每处 `文件:行号` 锚点跟 HEAD 逐一核对——**几乎零漂移**（仅 `getTableSchema` @Override 签名上移 10 行、方法体 419-440 不变）；**定论唯一开放问题**——预编译 EXECUTE 复用同一 `StatementContext`，重置钩子应挂 `ExecuteCommand:90-91`（那两行既有 per-execution 重置旁），**不**挂通用 setter（`setConnectContext` 11 caller 无一在语句中途、但过宽）；作用域 key 含 queryId 为第二道防线。另修正设计两处笔误（`ConnectorSession` 实现实为 21 非 14，仅生产 `ConnectorSessionImpl` 改；删除面比设计列的多——扫描 4 + 写 2 个 ctor、两测试类调用点、两处 `{@link}` 悬空）。 +- **用户拍板**：写路径口径矛盾（任务清单旧写"beginWrite 保持新载" vs 权威设计"取共享表"），上交 AskUserQuestion → **选完整统一版**（读写共享一次加载 + 拆胖句柄 + stash 下沉）。 +- **实现（两个 `[perf]` commit）**： + - **fe-core 基础**（`97bdcd6bdbe`）：`fe-connector-api` 加 `ConnectorStatementScope`(接口+NONE)+ `ConnectorSession.getStatementScope()` 默认；`fe-core` 加 `ConnectorStatementScopeImpl`(CHM) + `StatementContext` 懒建字段/同步访问器/`resetConnectorStatementScope`(不在 close/release 清、随 GC，镜像 snapshots) + `ConnectorSessionImpl/Builder` **构造期捕获**(from(ctx) 优先、回退 ConnectContext.get()、两级 null→NONE) + `ExecuteCommand` 一行重置。 + - **iceberg**（`ea7fd1f6e7a`）：`IcebergStatementScope` helper(`sharedTable` 键 catalogId:db:tbl:queryId + `rewritableDeleteSupply` 键 catalogId:queryId、null-session 兜底)；读 `resolveTableForRead`(+session)/扫描 `resolveTable`/写 `resolveTable`/`beginWrite` 四处走它；**拆 L0**(删 `resolvedTable` 字段/访问器/三 with* 携带 + 两 seam 读写)；`beginWrite` 取共享表(保留 openTransaction refresh)；扫描 accumulate + 写 drain 经作用域同键 map，**整删** `IcebergRewritableDeleteStash`(141 行)+ 测试(204 行)+ 六 ctor 参数；写侧 v3 DELETE/UPDATE/MERGE + NONE **fail-loud**。 +- **关键取舍**：`beginWrite` 保留 refresh（`Transactions.newTransaction` 本就 refresh；保留更安全=新鲜 OCC 基底 + 消解「共享表比 pin 旧」跨缓存错位；读扫描显式钉快照读不可变数据、不受共享对象被 refresh 到 latest 影响）；写侧性能收益≈0，真交付=架构连贯 + 删单例/类。 +- **测试**：iceberg 重写 3 个扫描 stash 测试 + 5 个写 stash 测试为作用域版 + 新增 fail-loud/度量守门/`IcebergStatementScopeTest`(键隔离)/`TestStatementScope` 助手；删 2 个 L0 handle 测试 + `IcebergRewritableDeleteStashTest`。fe-core 新增 `ConnectorStatementScopeTest`(NONE 不 memo/impl memo+隔离/StatementContext 懒建+reset)。 +- **踩坑**：`planScan(null,...)` 空 session 测试 → `sharedTable`/`rewritableDeleteSupply` 加 null 兜底(等价 NONE)；删 L0 测试后 `IcebergTableHandleTest` 5 个 import 悬空(checkstyle 挂 test 源)→ 清理。 +- **对抗复审**：6 视角多 agent(读路径/凭证隔离/快照OCC/删除供给/SPI捕获/L0删除)+ 逐条对抗核实 → **0 确认发现**（3 lens 首轮 stream timeout，单独重跑亦全 clean）。凭证隔离核实：读/扫描/写均 `newCatalogBackedOps(session)` 同用户 + 作用域每语句(不同 ConnectContext→不同 StatementContext→不同 map)；删除供给核实：扫描/写同 StatementContext scope 同 key、parity 逐字、fail-loud 恰覆盖消费 rewritableDeletes 的三 op。 +- **结果**：iceberg **968 pass / 0 fail / 1 skip**；fe-core 两段验 `ConnectorStatementScopeTest` 3 + `ConnectorSessionImplTest` 17 绿；两处 BUILD SUCCESS + checkstyle 0 违规，0 回归。summary 见 `designs/FIX-PERF-07-unified-per-statement-table-owner-summary.md`。 +- **新判据（可复用）**：**「唯一贯穿语句的对象是 `StatementContext`」**——连接器 session 一语句被重建 ~26 次、缓存挂它即死；跨读写共享须向上够到 StatementContext(经中性 SPI `getStatementScope()`)，**构造期捕获**(非实时读)才能让 off-thread 扫描泵够到(它们复用请求线程建的同一 session、无 ConnectContext thread-local)。 +- **下一步**：见 HANDOFF —— PERF-08（C19/C21 维护路径逐单位重扫/无去重；改动小可先行）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index f832f89caf0095..6b004c1e1bbc92 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -21,7 +21,7 @@ | PERF-04 | P1 | C17 C18 | streaming / COUNT(*) 下推旁路 IcebergManifestCache → 抽惰性 `cacheBackedFileScanTasks`(delete 索引 eager + data manifest 惰性扁平映射)三处复用;缓存与不 OOM 兼得(否决审计"退回物化"因重引 OOM) | — | ✅ 完成 | `2e5f393779c` | | PERF-05 | P1 | C9 | information_schema.tables 每表 loadTable 取 comment → 复核缩小(普通目录 PERF-01 已覆盖);补**无 gate** `IcebergCommentCache` **仅 vended 非 session=user**(红队:session=user 缓存绕过 per-user 授权=泄漏) | — | ✅ 完成 | `aea3ebdd40e` | | PERF-06 | P1 | C3 | REST vended-cred 每 data/delete file 重建 StorageProperties+Configuration → scan 级「准备一次、逐文件套用」normalizer(SPI seam+fe-core override,连接器侧 hoist);~~fe-core 框架 memo~~否决(per-catalog 跨查询共享→并发冲刷退化+凭证保留贴红线) | — | ✅ 完成 | `6294edf2833` | -| PERF-07 | P1 | C20 | 一条 DML 3~5 次 load 同表 → 语句级 resolve 一次传递 | — | ⏳ | | +| PERF-07 | P1 | C20 | 一条 DML 重复 load 同表 → 每语句「表加载归属者」(`ConnectorStatementScope` 挂 StatementContext) 读写共享一次加载；拆胖句柄；delete stash 下沉到每语句作用域(删单例+类)；beginWrite 取共享表 | — | ✅ 完成 | `97bdcd6bdbe`+`ea7fd1f6e7a` | | PERF-08 | P2 | C19 C21 | 维护路径逐单位重扫 / 无去重（rewrite_data_files 每 group planFiles；expire_snapshots S×M）→ union 一次注册 / 按 path 去重 | 改动小可先行 | ⏳ | | | PERF-09 | P2 | C5 | **（fe-core 框架层）** streaming pump 逐 split 重建 backend 候选集 + 锁往返 → 微批 | ⚠ 跨连接器，见约束 | ⏳ | | | PERF-10 | P2 | C8 | 同组 WHERE conjunct 被转换 5~6 次/查询 → node 字段 memo（与 01 同点失效） | 与 01 同源 | ⏳ | | @@ -71,9 +71,13 @@ - **落地**：三处 scan-start（同步/流式/position_deletes）构造一次 normalizer 往下传，六个 per-file seam 参数由 token 换成 normalizer；`TcclPinning` 透传拿到提升；写路径 `getBackendFileType`（O(1)/写）不动。惰性 memo 对齐原逐文件版空URI短路/vended覆盖/坏路径 fail-loud/异常时机。 - **收益/守门**：一次 vended scan 内 `buildVendedStorageMap`/`createAll` 从 O(N_files+N_deletes)→1。连接器守门测试断言 3 文件 scan `newNormalizerCount==1` 且 `normalizeCount==3`；fe-core parity 测试断言逐字等价。iceberg 974 UT 绿 + fe-core NormalizeUriTest 13 绿。 -### [ ] PERF-07 — C20：写路径 3~5 次 load 同表 -- **病灶**：`PhysicalIcebergMergeSink.getRequirePhysicalProperties:161→198`、`PhysicalPlanTranslator.visitPhysicalConnectorTableSink:675,703`、`PluginDrivenTableSink.bindDataSink:175 → IcebergWritePlanProvider.resolveTable:689-702 / beginWrite`（内含 tableExists×2 + 无条件 refresh）。每条 DML +3~5 次串行远程往返。 -- **修复方向**：语句级 resolve 一次传递；exists 从 load 结果推导。 +### [x] PERF-07 — C20：一条 DML 重复 load 同表 · ✅ `97bdcd6bdbe`(fe-core)+`ea7fd1f6e7a`(iceberg) +- **权威设计/小结**：[`designs/FIX-PERF-07-unified-per-statement-table-owner-design.md`](./designs/FIX-PERF-07-unified-per-statement-table-owner-design.md) + [`-summary.md`](./designs/FIX-PERF-07-unified-per-statement-table-owner-summary.md)；架构结论落 memory `iceberg-table-resolution-cache-scoping`。 +- **病灶**：读元数据/扫描/写成形/`beginWrite` 各自 loadTable 同表（最糟一条 DML 3~5 次）；旧两层缓存（胖句柄 + 跨查询 `IcebergTableCache`）只覆盖读扫描一段，写臂全绕。 +- **方案（用户 2026-07-18 拍板「完整统一版」，取代旧窄设计与审计原方向）**：新增中性 `ConnectorStatementScope`（挂 `StatementContext`、经 `ConnectorSession.getStatementScope()` 够到、构造期捕获以跨 off-thread）作**读写共享**的每语句表加载归属者；读/扫描/写/`beginWrite` 四处全走它，整条语句一张表**只加载一次**；**拆胖句柄**（句柄回归纯坐标）；**delete stash 下沉到每语句作用域**（整删 `IcebergRewritableDeleteStash` ~140 行 + 测试 + 两 provider 六个 ctor 参数）；v3 行级 DML + NONE 作用域**响亮失败**（杜绝静默复活）。`beginWrite` **取共享表**（保留 openTransaction 的 refresh 兜新鲜 OCC 基底）。 +- **⚠ 铁律**：净增 fe-core SPI（碰铁律 A，用户已签字）；prepared-stmt 复用 StatementContext→scope 每执行重置（`ExecuteCommand`）。 +- **验证**：iceberg 968 pass / 0 fail / 1 skip；fe-core 两段验（`ConnectorStatementScopeTest` 3 + `ConnectorSessionImplTest` 17 绿）；6 视角多 agent 对抗复审 0 确认发现。e2e 留 P6.6 切换阶段补。 +- **审计原「exists 从 load 推导 / resolve 一次跨层传递 / beginWrite 保持新载」均已作废**。 --- From f6a63a548a63ccb481b3f22482ec43fa77c46317 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 20:52:55 +0800 Subject: [PATCH 19/47] [doc](catalog) fe-connector-iceberg PERF-07: add the authoritative design docs Commit the two design docs the tracker/summary reference: the unified per-statement table-owner design (authoritative) and the superseded narrow resolution-owner design it replaces. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- ...7-per-statement-resolution-owner-design.md | 77 ++++++++ ...nified-per-statement-table-owner-design.md | 164 ++++++++++++++++++ 2 files changed, 241 insertions(+) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-per-statement-resolution-owner-design.md create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-design.md diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-per-statement-resolution-owner-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-per-statement-resolution-owner-design.md new file mode 100644 index 00000000000000..f67d85b4c90a5b --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-per-statement-resolution-owner-design.md @@ -0,0 +1,77 @@ +# FIX-PERF-07 设计 — 每语句解析 owner（StatementContext-hosted）+ delete stash 下单例 + +> 本设计**取代**审计原方案（"语句级 resolve 一次传递 / connector 侧 queryId memo 挂单例"）。 +> 经 3 轮对抗 workflow + 用户 2 次拍板收敛，结论落 memory `iceberg-table-resolution-cache-scoping`。 + +## 用户已定的两个前提（硬约束） + +1. **`beginWrite` 保持新载**（fresh + `newTransaction().refresh()`）——它是 OCC 锚点，构造上永不进 memo。 +2. **走"乙"**：真正把 delete stash 从 `IcebergConnector` 单例挪走（不是在单例上换个 queryId 记忆），用户已对"净增 fe-core SPI（碰铁律 A）"签字。 + +## 病灶 & 诚实的价值定位 + +- 写路径 `resolveTable`（`IcebergWritePlanProvider:689`）裸 loadTable、无 memo，`getWriteSortColumns`/`getWritePartitioning`/`appendExplainInfo` 各自加载；`beginWrite`（`IcebergConnectorTransaction:208`）再新载。 +- **但**：`beginWrite` 保持新载后写侧性能收益 ≈ 0（派生在普通目录也命中 tableCache，仅 gated-catalog + EXPLAIN 有薄收益）。 +- **真正价值 = 架构连贯 + stash 下单例**（原则性，非修 bug；现 stash 工作正常、TTL 兜漏从不触发）。 +- **读热缝不动**：`resolveTableForRead` / 扫描侧 resolve 是 PERF-01~06 刚稳的代码，铁律 3 保护，全高度（重写读缝、删 fat-handle、tableCache 下沉 CachingCatalog）留**远期目标**。 + +## 设计 — 一个中性的每语句 scope + +**对象模型**：新增中性 SPI 类型 `ConnectorStatementScope`（`fe-connector-api`，连接器不碰 fe-core 类型），一个方法 + no-op 单例： +```java +interface ConnectorStatementScope { + T computeIfAbsent(String key, java.util.function.Supplier loader); + ConnectorStatementScope NONE = new ConnectorStatementScope() { + public T computeIfAbsent(String k, Supplier l) { return l.get(); } // 不缓存=与今天逐次加载字节一致 + }; +} +``` +**物理 home** = `StatementContext` 上一个懒建字段（**不用** `getOrRegisterCache`——其"首个调用者拿到的实例≠后续"的坑对可变容器是正确性 bug）。普通语句随 StatementContext GC（`ConnectContext.clear()` 置空）。⚠ **prepared-statement EXECUTE 复用同一个 StatementContext**（`ExecuteCommand:89` 取 `preparedStmtCtx.getStatementContext()`，prepared INSERT/UPDATE 亦然）——故 scope **必须在每次 EXECUTE 边界显式重置**（否则 N 次执行累积 N 个条目直到 prepared 释放；这正是现 stash 的 TTL sweep 在处理的场景，先前"从不触发"判断有误）。重置钩子挂在 StatementContext 复用点（`setConnectContext`/prepared 重指向处，iceberg-agnostic 的一行 `connectorStatementScope=null`）；确定性重置优于 TTL。key 里的 queryId 是第二道防线（每次执行 queryId 新→旧条目在新 queryId 下不可达，杜绝跨执行脏读）。 +```java +private ConnectorStatementScope connectorStatementScope; // 镜像 getOrCacheDisableRules 的同步懒建 +public synchronized ConnectorStatementScope getOrCreateConnectorStatementScope() { ... } +``` +**够到方式** = `ConnectorSession.getStatementScope()`（新增默认方法，返回 `NONE`）；`ConnectorSessionImpl` override 返回构建时从 `ConnectContext.get().getStatementContext()` 捕获的引用（两级 null → NONE，镜像 `MvccUtil.getSnapshotFromContext`）。一条语句 ~26 个 session 实例引用**同一个** scope（身份在 StatementContext 上）。 + +**freshness 契约 = 方法身份**（非可默认错的布尔）：`resolveTableForRead`(读,可命中缓存) / 写 `resolveTable`(只 memo,无 tableCache) / `beginWrite`(构造上 fresh,无 handle/memo 引用)。 + +## 连接器改动（iceberg） + +1. **`IcebergWritePlanProvider.resolveTable`（:689-702）**：现有 body 包进 + `session.getStatementScope().computeIfAbsent("iceberg.write.table:"+catalogId+":"+db+":"+tbl+":"+queryId, () -> <现 body>)`。 + 保留 `catalogOpsResolver.apply(session)` + `executeAuthenticated` + `context==null` 分支不变。收敛 3 个派生调用；`beginWrite` **不改**。 + > ops-source 安全：memo 只被写路径填/取，全程同一 per-session ops、同一用户；读路径不经此 scope，无跨 ops-source 复用。 +2. **stash 迁移**： + - 删 `IcebergConnector.rewritableDeleteStash` 字段（:185）+ getScan/getWritePlanProvider 的 ctor 参数（:680-682,:691-693）。 + - **整删 `IcebergRewritableDeleteStash.java`（~140 行）+ 其测试**（queryId 键、Entry、TTL sweep 全被每语句 GC 收编）。 + - 扫描侧（`IcebergScanPlanProvider` ~:679-680,:767）：`formatVersion>=3` 时经 scope 取一张 + `computeIfAbsent("iceberg.rewritable-delete-supply:"+catalogId+":"+queryId, ConcurrentHashMap::new)`，per range `put(rawPath, descs)`（同 raw-path 键、同 empty 守卫）；流式路径 `stashRewritableDeletes=false` 仍不触碰。 + - 写侧（`IcebergWritePlanProvider.planWrite` :191-192）：`session.getStatementScope().computeIfAbsent(同键, ConcurrentHashMap::new)` 直接读，`buildRewritableDeleteFileSets` 不变（空 map→unset thrift 字段，字节一致）。无需 remove（随语句 GC）。 +3. **fail-loud 断言**：`formatVersion>=3 && (DELETE||UPDATE||MERGE) && getStatementScope()==NONE` → 抛 `DorisConnectorException`（把唯一新增的"静默复活"降为响亮失败）。 + +## fe-core 改动（净增，碰铁律 A——用户已签字） + +- `fe-connector-api`：`ConnectorStatementScope.java`（接口+NONE）；`ConnectorSession.java` 加一个默认方法。 +- `fe-core`：`ConnectorStatementScopeImpl.java`（`ConcurrentHashMap`）；`StatementContext` 加懒建字段+同步访问器（**不在 close()/release 清，随对象 GC**，镜像 snapshots map）；`ConnectorSessionImpl` 加字段+override；`ConnectorSessionBuilder` 两级 null 捕获 + `withStatementScope` 测试钩子。 +- **读路径 fe-core 代码零改动**；连接器 SPI 14 个实现零改动（默认方法）。 + +## 正确性 / parity + +- `beginWrite` 不变 → OCC 锚点、分布式 rewrite（writeStarted 只让首 group 加载）字节一致。 +- 派生只读 `spec()/sortOrder()/schema()`（快照不变量），memo 存 RAW 表、认证取用时套 → parity。 +- **线程**：v3 DELETE/MERGE 的扫描累积 + 写抽取都在 StmtExecutor 请求线程、扫描先于写入 → 同一 StatementContext；分布式 rewrite 各 group 自建 StatementContext（碎片化）但 REWRITE 不用 stash → 无害；读 off-thread split 池不经 scope（走 fat-handle）。 +- **跨 catalog MERGE**：key 含 catalogId → 复刻旧 per-catalog 隔离；raw-path 全局唯一无碰撞。 + +## 测试守门 + +- 单测：计数 `IcebergCatalogOps` + 真 memoizing 测试 scope，断言 EXPLAIN INSERT 派生 loadTable 2→1；NONE 对照=逐次；stash parity（同 catalogId+queryId 的 scan/write session 共享一张 map，v3 DELETE 抽出恰好那些 set）；跨 catalog 隔离；fail-loud 触发；`beginWrite` 仍新载 + startingSnapshotId=begin 时快照。改 `IcebergScan/WritePlanProviderTest` 去掉 stash ctor 参数；删 `IcebergRewritableDeleteStashTest`。 +- e2e（P6.6 cutover 补，对齐 `hms-iceberg-delegation-needs-e2e`）：独立 + HMS 网关目录 INSERT/DELETE/MERGE/OVERWRITE 无复活；分布式 rewrite；跨 catalog MERGE 只抽目标 catalog 供给。 + +## 待验（落地前） + +- **prepared-statement 复用 StatementContext 已确认**（`ExecuteCommand:89`, `StmtExecutor:268-272`）→ scope 须每次 EXECUTE 重置（见"物理 home"）。落地前须确认**重置钩子挂点**：`StatementContext.setConnectContext`/复用重指向处是否只在语句/执行边界调用（grep 其 caller，确保不在语句中途调用而误清活 scope）；若不稳，改为在 `ExecuteCommand` 显式重置 + 普通语句天然新建两条路各自保证。 + +## 遗留（未做，记录） + +- 每条 DML 的 1~2 次 `tableExists`（走通用 `getTableHandle`）不动（读/DDL/元数据共用，改动面大）。 +- 读 gated-catalog 冗余（session=user/vended 各站点各加载）不修——需路由读经 owner、动 PERF-01~06 热缝，全高度远期目标。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-design.md new file mode 100644 index 00000000000000..5dda994cd8b819 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-design.md @@ -0,0 +1,164 @@ +# FIX-PERF-07（重定范围）设计 —— 每语句「表加载归属者」，读写共享 · 快照一致 + +> **本设计取代** [`FIX-PERF-07-per-statement-resolution-owner-design.md`](./FIX-PERF-07-per-statement-resolution-owner-design.md)（那版是窄范围：只 memo 写侧 resolveTable + 迁移删除清单、读路径不动、beginWrite 保持新载）。 +> **用户 2026-07-18 拍板走「完整统一版」**：要一个贯穿读写的每语句对象，表只加载一次、整条语句复用、快照一致；明确不追求性能、不介意净增 SPI，目标是架构更清晰合理。并拍板**拆掉胖句柄**。 +> 结论/取证：6 路侦察 workflow（`scratchpad/u-*.md`）+ Trino 源码逐条核实。相关记忆 `iceberg-table-resolution-cache-scoping`（待本设计落地后更新）。 + +--- + +## 0. 目标与非目标 + +**目标** +1. 一个挂在 `StatementContext` 的每语句作用域，作为整条语句（读 + 写）的表加载归属者。 +2. 语句涉及的每张表只加载一次（RAW `Table`），读元数据 / 扫描 / 写成形（排序列、分区）/ `beginWrite` 全部复用同一次加载。 +3. 保持既有快照一致性（S_read 钉读扫描 + 写提交 OCC），并让读写共享**同一个已加载表对象**，使全语句所有元数据派生自一次加载。 + +**非目标 / 明确不做** +- 不追求性能提升（写侧收益≈0；真实价值=架构连贯 + 删单例/删类）。 +- 不改快照钉选机制（`StatementContext.snapshots` / `IcebergLatestSnapshotCache` / `applyMvccSnapshotPin` 不动）。 +- 不把 sys 表加载、DDL 一次性加载并入登记处（保持新载）。 +- 不删跨查询缓存 `IcebergTableCache`（它服务**跨语句**复用，登记处覆盖不了）。 + +--- + +## 1. 现状（侦察确认） + +**三个各自为政的表解析器 + 写事务各自加载**（file:line 现值）： +- (A) 读元数据 `IcebergConnectorMetadata.resolveTableForRead:612-623`：胖句柄 → 跨查询缓存 → `loadTable`。 +- (B) 扫描 `IcebergScanPlanProvider.resolveTable:2202-2224`：同两层（memo 读 2206、`loadRawTable` 2231-2237），出口再 `wrapTableForScan`（Kerberos doAs FileIO）。 +- (C) 写 `IcebergWritePlanProvider.resolveTable:689-702`：**不查任何缓存，每次新载**。三个派生调用 `appendExplainInfo:241` / `getWriteSortColumns:257` / `getWritePartitioning:286` 各自触发。 +- (D) `IcebergConnectorTransaction.beginWrite:193-222`：又一次 `catalogOps.loadTable:208`（存 `transaction.table`）。 + +**两层既有缓存**： +- L0 胖句柄 `IcebergTableHandle.resolvedTable`（`transient`，字段 :113，访问器 195/200，`with*` 拷贝 217/232/243 携带）——只覆盖扫描节点那一段（每个 fe-core 入口各建 throwaway 句柄，不跨 lineage 桥接）。 +- L1 跨查询 `IcebergTableCache`（挂长生命周期 `IcebergConnector`，键 `TableIdentifier(db,table)`，存 RAW 表）——**对 `session=user` 与 REST vended 关闭**（`IcebergConnector.java:216-220`，凭证隔离）。 + +**审计「一次查询 3~7 次 loadTable」**由 throwaway-handle + 写侧全不缓存共同造成；写臂最糟（(C)+beginWrite+procedure/DDL 全绕缓存）。 + +**快照 S_read 机制（保持不变）**：`beginQuerySnapshot:1615-1627` 经 `IcebergLatestSnapshotCache` 收敛成 `(snapshotId, schemaId=最新)`，存 `StatementContext.snapshots:272`；`applyMvccSnapshotPin` 把它钉进读句柄和写句柄（`IcebergTableHandle.getSnapshotId()`）；读扫描 `useSnapshot(S_read)`（`buildScan:1022-1047`）、写 `baseSnapshotId=ctx.getReadSnapshotId()`（`applyBeginGuards:279-283`）→ `validateFromSnapshot(S_read)`。**RAW 表不冻结在 S_read**，pin 是每扫描/每提交再套。 + +--- + +## 2. Trino 对照（源码已核） + +- 每事务一个 `IcebergMetadata`（`IcebergTransactionManager` 的 `MemoizedMetadata`）；一条 SQL = 一事务。 +- 其 `TrinoHiveCatalog.tableMetadataCache`（键 SchemaTableName → TableMetadata）→ 一事务内 `loadTable` 只回一次远端，读规划与写共享同一 `TableMetadata`。 +- 快照在 `getTableHandle` 钉进 `IcebergTableHandle.snapshotId`，穿进读 `useSnapshot` 与写 `RowDelta.validateFromSnapshot`。 +- `beginInsert` 里 `catalog.loadTable(...)` **命中 tableMetadataCache** → 写事务从读期同一张表构建；提交 OCC 锚定读快照；iceberg 内部 CAS+重试吸收良性并发，`ValidationException` 硬失败、不自动重规划。 + +**Doris 无「每事务 metadata」**：连接器 session 一条语句被重建 ~26 次，缓存挂它即死。唯一贯穿语句的对象是 `StatementContext` → 连接器**向上够**到它（`ConnectorSession.getStatementScope()`）。这是 Trino 模型可移植的落点。 + +--- + +## 3. 设计 + +### 3.1 每语句作用域（中性 SPI） + +- `fe-connector-api` 新增 `ConnectorStatementScope`： + ```java + interface ConnectorStatementScope { + T computeIfAbsent(String key, java.util.function.Supplier loader); + ConnectorStatementScope NONE = -> l.get(); // 不缓存 = 与今天逐次加载字节一致 + } + ``` +- `ConnectorSession` 加默认方法 `default ConnectorStatementScope getStatementScope() { return ConnectorStatementScope.NONE; }`（14 实现零改）。 +- `fe-core`：`ConnectorStatementScopeImpl`（`ConcurrentHashMap` 背书）；`StatementContext` 加懒建字段 + `synchronized` 访问器（镜像 `getOrCacheDisableRules:632`；**不在 `close()`/`releasePlannerResources:917-948` 清**，随 GC，镜像 `snapshots`）。 +- **捕获方式 = 构造期捕获（非实时读）**：`ConnectorSessionImpl` 在**构造时**从 `ConnectContext.get().getStatementContext()` 取作用域引用（两级 null → `NONE`，镜像 `MvccUtil.getSnapshotFromContext:35-45`）。 + - **为何必须构造期捕获**：扫描流式/分批切分在 off-thread 线程池跑，那里 `ConnectContext.get()`（thread-local）可能为空；`PluginDrivenScanNode` 只在请求线程建**一个** session（字段 :148，`buildConnectorSession` :200）供异步任务复用（:1564）。构造期（请求线程）捕获 → session 带着作用域引用 → 任何线程都够得到。**实时读方案在 off-thread 失效，已否决。** + +### 3.2 表登记处（读写归一） + +作用域里一张登记处：键 `"iceberg.table:" + catalogId + ":" + db + ":" + tbl + ":" + queryId`，值 **RAW** `Table`。 +- **含 queryId**：预编译多次执行各自 queryId → 天然隔离（每次执行看当次的表）。 +- **不含 snapshot/ref**：RAW 表快照无关（对齐 L0/L1），pin 在下游每扫描/每提交套。 + +统一解析器（新增一个连接器私有 helper，姑且叫 `resolveTableShared(session, handle)`）： +``` +scope = session.getStatementScope() +raw = scope.computeIfAbsent(tableKey, () -> + L1 != null ? L1.getOrLoad(id, () -> executeAuth(ops.loadTable(db,tbl))) + : executeAuth(ops.loadTable(db,tbl))) +return raw // 消费者各自再套 wrapTableForScan / openTransaction / 提取 vended token +``` +- (A)(B)(C) 与 `beginWrite` 全部改走它。 +- 层次：登记处（每语句、所有授权模式开）→ L1（跨语句、仅普通目录）→ 终端 `loadTable`。 +- **拆掉 L0**（用户拍板）：删 `IcebergTableHandle.resolvedTable` 字段 + 访问器 + `with*` 三处携带 + (A)(B) 里的 memo 读写。句柄回归纯坐标。`NONE`（离线/无上下文）下 `computeIfAbsent` 每次调 loader = 与今天离线逐次一致。 + +### 3.3 `beginWrite` 改法（正面结论） + +- `beginWrite` **不再自己 `loadTable`**，改成 `table = resolveTableShared(session, handle)`（与读、算排序/算分区同一张）。 +- SDK 事务从这张共享表构建；**OCC 仍锚定 S_read**（`baseSnapshotId` 逻辑不变）；提交时 iceberg 自身 CAS + 内部重试对齐最新基底并校验 S_read 之后的冲突（良性并发重试吸收、真冲突硬失败、不自动重规划——与 Trino 同姿态）。**对齐 Trino `beginInsert` 命中读期缓存表来开事务。** +- 事务构造细节（keep 还是 drop `newTransaction()` 的隐式 refresh）为实现期决定项：优先**统一走 `Transactions.newTransaction(name, ops)` 从共享表 ops 直接建（不额外 refresh）**以最大化连贯（与 Trino 一致，提交 CAS 兜基底新鲜度）；以 parity 测试守门，若发现回归再退回保留 refresh。 +- 分布式 rewrite 的 `startingSnapshotId`（begin 时当前快照，非 S_read）语义**不变**——共享表的 `currentSnapshot()` 即该锚点，`registerRewriteSourceFiles` 仍在该快照 re-derive。 + +### 3.4 删除清单下沉 + +- 行级 DELETE/MERGE 的「可改写删除文件」清单从连接器单例挪进作用域：键 `"iceberg.rewritable-delete-supply:" + catalogId + ":" + queryId`，值 `Map>`。 +- 扫描侧 `buildRangeForTask`（accumulate 现在 :768，v3 gate :679-680；流式 hard-code false/null 不触碰）改成 `scope.computeIfAbsent(supplyKey, ConcurrentHashMap::new).put(rawPath, descs)`；写侧 `planWrite:191-192` 从同键 map 直接读。 +- **整删** `IcebergConnector.rewritableDeleteStash` 字段（:185）+ 两 provider ctor 参数（:680-682,:691-693）+ **整个 `IcebergRewritableDeleteStash.java`（141 行）+ 其测试**（queryId 分桶/TTL sweep 被每语句 GC + 每执行重置收编）。 +- **新增响亮失败**：`formatVersion>=3 && (DELETE|UPDATE|MERGE) && getStatementScope()==NONE` → 抛 `DorisConnectorException`（今天这里是静默返回空 → 迁移引入的唯一「静默复活」风险降为响亮失败）。放写侧 `planWrite`（此处已知 writeOperation + formatVersion）。 + +### 3.5 预编译重置 + +- `ExecuteCommand.run:89` 复用同一 `StatementContext`。在其已有 per-execution 重置（`setPrepareStage(false)`/`setIsInsert(false)`，:90-91）旁加一行**重置作用域**（`statementContext.resetConnectorStatementScope()`）。 +- 键含 queryId 为第二道防线（隔离），重置管内存上界；两者独立。普通语句每条新建 `StatementContext` 天然新作用域，无需依赖重置。 + +--- + +## 4. 授权 / 正确性 / parity + +### 4.1 授权安全（session=user / vended）——5 条不变式 +1. 作用域 session-bound、随语句生死，**永不**升进任何跨 session/表身份键的结构。 +2. 存 **RAW** 表；delegated/Kerberos FileIO、vended token 取用时逐消费者重套/重取，绝不冻进作用域。 +3. 一语句 = 一用户 = 一凭证；读写用同一 session 解析 ops → 无第二身份可泄漏。 +4. fail-closed 保持：ops 在 auth scope 前解析，tokenless `session=user` 抛错，作用域绝不被 fallback 共享加载污染。 +5. vended token 语句内新鲜：作用域不跨语句 → 无过期风险。 +→ 故登记处对三种授权模式**都能全开**（正是 L1 对凭证目录关闭、此处能开的原因），且凭证目录（L1 关）受益最大。 + +### 4.2 快照一致性 +- 「删哪些行/冲突检测/改写哪些旧删除」今天已全钉 S_read（`baseSnapshotId` + `validateFromSnapshot` + `collectRewrittenDeleteFiles` 读 baseSnapshotId 删除清单）——登记处**不碰**。 +- **诚实修正**：写新文件用**最新** schema/spec 是 iceberg 有意语义（非 bug）；对普通 DML，S_read 的 schema == 最新（捕获时取最新 schemaId）。统一的价值是**读写共享同一已加载表对象、全语句派生自一次加载 = 连贯**，不是修 write-shaping bug。 +- **schema-dict 不变式**：BE 字段 ID 字典 `pinnedSchema:1064-1073` 与 FE schema 元组 `getTableSchema(...,snapshot):419-440` 必须解析同一 schemaId（否则 schema 演进的时间旅行读 BE `children.at()` SIGABRT）。登记处集中加载后两侧仍读同一 RAW 表的 `schemas().get(schemaId)`，字节不变。 + +### 4.3 线程 +- 读元数据/同步扫描/写全在请求线程；流式 `streamSplits`、分批 `planScanForPartitionBatch` 在 off-thread 池，复用请求线程建的 session → 构造期捕获保证够到作用域。iceberg `ParallelIterable` 只读已解析 `table.io()`，不新 loadTable。 + +--- + +## 5. fe-core 净增清单（碰铁律 A，用户签字） + +- `fe-connector-api`：`ConnectorStatementScope.java`（接口+NONE）；`ConnectorSession.getStatementScope()` 默认方法。 +- `fe-core`：`ConnectorStatementScopeImpl.java`；`StatementContext` 懒建字段+同步访问器+`resetConnectorStatementScope()`；`ConnectorSessionImpl` 构造期捕获+override；`ConnectorSessionBuilder` 传递捕获；`ExecuteCommand` 重置一行。 +- 读路径 fe-core 逻辑零改（仅 SPI 默认方法 + session 捕获）。 + +--- + +## 6. 实现顺序 + +1. `fe-connector-api`：`ConnectorStatementScope`（接口+NONE）+ `ConnectorSession.getStatementScope()`。 +2. `fe-core`：`ConnectorStatementScopeImpl` + `StatementContext` 字段/访问器/重置 + `ConnectorSessionImpl`/`Builder` 构造期捕获 + `ExecuteCommand` 重置。fe-core 单测。 +3. iceberg：`resolveTableShared` helper；(A)(B)(C)+`beginWrite` 接它；**拆 L0**（删胖句柄字段/访问器/携带/memo 读写）；`beginWrite` 取共享表 + 事务从共享表构建。iceberg 单测。 +4. iceberg：删除清单下沉（扫描 accumulate + 写 drain 经作用域同键 map）；fail-loud；删 stash 类+单例字段+ctor 参数+测试。iceberg 单测。 +5. 度量守门 + 独立 commit（可拆 2~3 个）+ 写 summary + 更新 tasklist/HANDOFF/memory。 + +--- + +## 7. 测试守门 + +- **加载计数**：给终端加载加 `loadCountForTest()`（今 table 路径缺此钩子）；断言一条读+写语句对同一表 3~7→1；`NONE` 对照=逐次。 +- **读写共享**：读扫描与写 `beginWrite` 拿到同一 `Table` 实例（identity 断言）。 +- **凭证目录**：session=user / vended（L1 关）下仍归一。 +- **快照**：`beginWrite` 仍锚定 S_read；schema 演进时间旅行读 FE/BE schemaId 一致。 +- **删除清单 parity**：同 catalogId+queryId 扫描/写共享一张 map，v3 DELETE 抽出恰好那些 set；跨 catalog MERGE 隔离；fail-loud 触发。 +- **预编译**：重执行不累积（重置生效）+ 跨执行隔离（queryId）。 +- fe-core 侧改动另跑 fe-core 单测（iceberg 反应堆不含 fe-core，须两段验）。 +- e2e（后续切换阶段补，对齐 `hms-iceberg-delegation-needs-e2e`）：独立+HMS 网关目录 INSERT/DELETE/MERGE/OVERWRITE 无复活 + 分布式 rewrite + 跨 catalog MERGE。 + +--- + +## 8. 风险 / 待验 + +- **拆 L0 触碰刚稳定的读热路径**：靠加载计数 + 共享 identity + off-thread 归一测试守门；off-thread 归一依赖构造期捕获（已核 session 复用）。 +- **`beginWrite` 事务构造改动**（drop refresh vs keep）：以 parity 测试守门，倾向 drop refresh 对齐 Trino，回归则退回 keep。 +- **行号信 grep**（PERF-06 后可能又漂）。 +- **构建/验证坑**照 HANDOFF「构建/验证坑」小节（绝对 -f、`-am`、`install`、`-Dmaven.build.cache.enabled=false`、别 `-q`、别 `nohup &` 套后台）。 From c6376dae7925a607ed01609666d00f7a5aa7d2d5 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 22:20:49 +0800 Subject: [PATCH 20/47] [perf](catalog) fe-core: collapse rewrite_data_files per-group registration into one union scan (PERF-08 / C19) Root cause: ConnectorRewriteDriver STEP3 registered each bin-packed rewrite group's source files with a separate connectorTx.registerRewriteSourceFiles call. The iceberg connector re-derives those raw paths back to DataFiles with a full-table planFiles() scan on every call, all against the SAME once-pinned OCC snapshot (startingSnapshotId, captured at beginWrite). So G groups triggered G+1 full-table manifest scans; at G ~ 50-200 that is minutes of pure planning IO. Fix: union every group's paths into one Set and register once (unionSourceFilePaths). registerRewriteSourceFiles already accepts a Set and accumulates across calls, and every per-group call scanned the identical pinned snapshot, so one union scan reconstructs exactly the same filesToDelete set: bin-packed groups are path-disjoint and the connector dedups by path, so no double-registration; the four result counters come from group metadata, not filesToDelete; any unresolvable path still fails loud + rolls back in both variants. G planFiles() scans collapse to 1. No change to result row, committed state, or success/failure outcome. Connector side unchanged. Constraint: fe-core edit only converges an existing driver's calls to an existing neutral SPI (Set, no source-specific branch, no logic moved into fe-core) -- within the "fe-core source only shrinks / stays connector-agnostic" rules. Gate: the distributed STEP3 needs a live cluster (deferred to the flip rehearsal e2e), so unionSourceFilePaths is extracted and unit-tested for union/cross-group dedup/empty-group/empty-plan parity. ConnectorRewriteDriverTest 5/5 green; 0 checkstyle violations. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../execute/ConnectorRewriteDriver.java | 32 ++++++++++++--- .../execute/ConnectorRewriteDriverTest.java | 41 +++++++++++++++++++ 2 files changed, 67 insertions(+), 6 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java index b85c672fb50b7b..87a982a073f2fe 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java @@ -42,8 +42,10 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.Set; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; @@ -137,12 +139,16 @@ public ConnectorProcedureResult run() throws UserException { // STEP 2: run one INSERT-SELECT per group concurrently, all sharing the transaction. runGroups(groups, txnId, connectorTx); - // STEP 3: register the union of source data files to remove. AFTER the groups ran, so the first - // group's write loaded the table + pinned the OCC snapshot that the connector re-derives against; - // BEFORE commit, which consumes the registered files in the RewriteFiles op. - for (ConnectorRewriteGroup group : groups) { - connectorTx.registerRewriteSourceFiles(group.getDataFilePaths()); - } + // STEP 3: register the UNION of every group's source data files in a SINGLE call. The connector + // re-derives them from the table at the pinned OCC snapshot with ONE planFiles() scan; the former + // per-group loop repeated that full-table scan once per group (G groups = G+1 scans). Ordering is + // unchanged — still AFTER the groups ran (so the first group's write loaded the table + pinned the + // OCC snapshot that the connector re-derives against) and BEFORE commit (which consumes the + // registered files in the RewriteFiles op). Every per-group call scanned the SAME pinned snapshot, so + // one union scan is equivalent; the connector's registration accumulates and dedups by path, and the + // planner emits path-DISJOINT groups (iceberg: planFiles() yields one task per data file, bin-packed + // into disjoint groups), so the union reconstructs exactly the per-group calls' accumulated file set. + connectorTx.registerRewriteSourceFiles(unionSourceFilePaths(groups)); } catch (Exception e) { txnManager.rollback(txnId); if (e instanceof UserException) { @@ -167,6 +173,20 @@ public ConnectorProcedureResult run() throws UserException { removedDeleteFilesCount); } + /** + * Unions every group's source data-file paths into one dedup'd set, so the connector re-derives them all in + * a single {@code planFiles()} scan (STEP 3) instead of one scan per group. Bin-packed groups are + * path-disjoint so this is a straight union; the connector's own per-path dedup keeps the registered file set + * exact regardless. Package-visible for unit testing (the full distributed STEP 3 needs a live cluster). + */ + static Set unionSourceFilePaths(List groups) { + Set sourceFilePaths = new HashSet<>(); + for (ConnectorRewriteGroup group : groups) { + sourceFilePaths.addAll(group.getDataFilePaths()); + } + return sourceFilePaths; + } + private void runGroups(List groups, long txnId, ConnectorTransaction connectorTx) throws UserException { List tasks = Lists.newArrayList(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriverTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriverTest.java index 6d32290f1cdeba..9a95ae23067e60 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriverTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriverTest.java @@ -26,12 +26,14 @@ import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.procedure.ConnectorProcedureOps; import org.apache.doris.connector.api.procedure.ConnectorProcedureResult; +import org.apache.doris.connector.api.procedure.ConnectorRewriteGroup; import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; import org.apache.doris.connector.api.pushdown.ConnectorPredicate; import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; import org.apache.doris.qe.ConnectContext; +import com.google.common.collect.ImmutableSet; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import org.mockito.ArgumentCaptor; @@ -40,6 +42,7 @@ import java.util.Arrays; import java.util.Collections; import java.util.List; +import java.util.Set; import java.util.stream.Collectors; /** @@ -123,4 +126,42 @@ public void planRewriteFailureSurfacesAsUserException() { Assertions.assertTrue(ex.getMessage().contains("plan boom"), "the connector failure text must be preserved, got: " + ex.getMessage()); } + + @Test + public void unionSourceFilePathsMergesAllGroupsAndDedupsByPath() { + // STEP 3 registers the UNION of every group's source files in ONE connector call (one planFiles() scan) + // instead of one call per group. Disjoint groups union straight; a path recurring across groups collapses + // to a single entry, so the connector never double-registers a file to delete. MUTATION: unioning only + // the first group (or not deduping) is killed here. + ConnectorRewriteGroup g1 = new ConnectorRewriteGroup( + ImmutableSet.of("s3://b/t/a.parquet", "s3://b/t/b.parquet"), 2, 2048L, 0); + ConnectorRewriteGroup g2 = new ConnectorRewriteGroup( + ImmutableSet.of("s3://b/t/c.parquet"), 1, 1024L, 0); + // Defensive: a path shared with g1 (bin-packing keeps groups disjoint, but the union must still dedup). + ConnectorRewriteGroup g3 = new ConnectorRewriteGroup( + ImmutableSet.of("s3://b/t/a.parquet", "s3://b/t/d.parquet"), 2, 2048L, 0); + + Set union = ConnectorRewriteDriver.unionSourceFilePaths(Arrays.asList(g1, g2, g3)); + + Assertions.assertEquals( + ImmutableSet.of("s3://b/t/a.parquet", "s3://b/t/b.parquet", "s3://b/t/c.parquet", + "s3://b/t/d.parquet"), + union, "the union must contain each distinct source path exactly once across all groups"); + } + + @Test + public void unionSourceFilePathsSkipsEmptyGroupsAndEmptyPlan() { + // An empty group contributes nothing; an all-empty plan unions to the empty set (the connector treats + // that as a no-op registration — the same net state as the former loop making N early-returning calls). + ConnectorRewriteGroup withFiles = new ConnectorRewriteGroup( + ImmutableSet.of("s3://b/t/a.parquet"), 1, 1024L, 0); + ConnectorRewriteGroup empty = new ConnectorRewriteGroup(Collections.emptySet(), 0, 0L, 0); + + Assertions.assertEquals(ImmutableSet.of("s3://b/t/a.parquet"), + ConnectorRewriteDriver.unionSourceFilePaths(Arrays.asList(withFiles, empty)), + "an empty group must not affect the union"); + Assertions.assertTrue( + ConnectorRewriteDriver.unionSourceFilePaths(Collections.emptyList()).isEmpty(), + "an all-empty plan unions to the empty set"); + } } From 39f8cb89164d1027674ebaec5bc9454ff68f6683 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 22:23:01 +0800 Subject: [PATCH 21/47] [perf](catalog) fe-connector-iceberg: dedup expire_snapshots delete-manifest reads across snapshots (PERF-08 / C21) Root cause: IcebergExpireSnapshotsAction.buildDeleteFileContentMap built the delete-file -> content classification map by looping over every snapshot and reading all of that snapshot's delete manifests, with no visited-path dedup. Iceberg manifests are immutable and adjacent snapshots carry the same delete manifests forward unchanged, so a single manifest is re-read once per referencing snapshot -- S snapshots x M manifests of serial remote reads, largely redundant. Fix: add a method-scoped visited set and skip a delete manifest whose path was already read (if (!visitedDeleteManifests.add(manifest.path())) continue). Reading each DISTINCT manifest exactly once collapses the reads from O(S*M) to O(distinct M). The result map is byte-identical: an immutable manifest yields the same DeleteFile set on every read, and putIfAbsent first-writer-wins is order- independent because a delete file's content type is immutable. visited MUST stay at method scope (a comment guards this) -- inside the snapshot loop it would reset each iteration and defeat the cross-snapshot dedup. Gate: new @VisibleForTesting lastDeleteManifestReadCount records the distinct manifests read; buildDeleteFileContentMap made package-visible. New test seeds a table whose delete manifest is carried across multiple snapshots (position + equality deletes), asserts reads == distinct manifests (strictly fewer than the per-snapshot total) and that both delete files stay classified as POSITION_DELETES / EQUALITY_DELETES. IcebergExpireSnapshotsActionTest 9/9 green; 0 checkstyle violations. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../action/IcebergExpireSnapshotsAction.java | 24 ++++++++- .../iceberg/action/ActionTestTables.java | 29 +++++++++++ .../IcebergExpireSnapshotsActionTest.java | 52 +++++++++++++++++++ 3 files changed, 104 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java index 4a3e63b0bc1abf..a10363ccc4c28c 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsAction.java @@ -24,6 +24,7 @@ import org.apache.doris.connector.api.pushdown.ConnectorPredicate; import org.apache.doris.foundation.util.ArgumentParsers; +import com.google.common.annotations.VisibleForTesting; import com.google.common.collect.Lists; import org.apache.iceberg.DeleteFile; import org.apache.iceberg.ExpireSnapshots; @@ -42,8 +43,10 @@ import java.time.format.DateTimeFormatter; import java.time.format.DateTimeParseException; import java.util.HashMap; +import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.Set; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.atomic.AtomicLong; @@ -67,6 +70,11 @@ public class IcebergExpireSnapshotsAction extends BaseIcebergAction { public static final String SNAPSHOT_IDS = "snapshot_ids"; public static final String CLEAN_EXPIRED_METADATA = "clean_expired_metadata"; + // Test-only gate for the delete-manifest dedup: the number of DISTINCT delete manifests read by the most + // recent buildDeleteFileContentMap call. Asserts each manifest is read once, not once per referencing snapshot. + @VisibleForTesting + int lastDeleteManifestReadCount; + public IcebergExpireSnapshotsAction(Map properties, List partitionNames, ConnectorPredicate whereCondition) { super("expire_snapshots", properties, partitionNames, whereCondition); @@ -268,8 +276,17 @@ private long parseTimestamp(String timestamp) { } } - private Map buildDeleteFileContentMap(Table icebergTable) { + @VisibleForTesting + Map buildDeleteFileContentMap(Table icebergTable) { Map deleteFileContentByPath = new HashMap<>(); + // Dedup delete-manifest reads across snapshots. Iceberg manifests are immutable and adjacent snapshots + // carry the same delete manifests forward unchanged, so re-reading one yields the identical DeleteFile + // set (putIfAbsent already made the re-read a no-op). Reading each DISTINCT manifest exactly once + // collapses the O(snapshots x manifests) remote reads to O(distinct manifests) with a byte-identical map. + // NOTE: visited MUST live at method scope (outside the snapshot loop) — inside it, it would reset every + // snapshot and defeat the cross-snapshot dedup this fix targets. + Set visitedDeleteManifests = new HashSet<>(); + int reads = 0; try { for (Snapshot snapshot : icebergTable.snapshots()) { List deleteManifests = snapshot.deleteManifests(icebergTable.io()); @@ -277,6 +294,10 @@ private Map buildDeleteFileContentMap(Table icebergTable) { continue; } for (ManifestFile manifest : deleteManifests) { + if (!visitedDeleteManifests.add(manifest.path())) { + continue; + } + reads++; try (CloseableIterable deleteFiles = ManifestFiles.readDeleteManifest( manifest, icebergTable.io(), icebergTable.specs())) { for (DeleteFile deleteFile : deleteFiles) { @@ -289,6 +310,7 @@ private Map buildDeleteFileContentMap(Table icebergTable) { } catch (Exception e) { throw new DorisConnectorException("Failed to build delete file content map: " + e.getMessage(), e); } + lastDeleteManifestReadCount = reads; return deleteFileContentByPath; } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/action/ActionTestTables.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/action/ActionTestTables.java index 10edf7bc70e5ac..fd30e387646f23 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/action/ActionTestTables.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/action/ActionTestTables.java @@ -22,6 +22,7 @@ import org.apache.iceberg.DataFile; import org.apache.iceberg.DataFiles; import org.apache.iceberg.FileFormat; +import org.apache.iceberg.FileMetadata; import org.apache.iceberg.PartitionSpec; import org.apache.iceberg.Schema; import org.apache.iceberg.Table; @@ -84,6 +85,34 @@ static DataFile dataFile(String fileName, long records) { .build(); } + /** Commits a position-delete file as a new snapshot (creates a delete manifest carried forward by later ones). */ + static void addPositionDeleteSnapshot(InMemoryCatalog catalog, String table, String fileName, + String referencedDataFile) { + catalog.loadTable(id(table)).newRowDelta() + .addDeletes(FileMetadata.deleteFileBuilder(PartitionSpec.unpartitioned()) + .ofPositionDeletes() + .withPath("s3://b/db1/" + fileName) + .withFormat(FileFormat.PARQUET) + .withFileSizeInBytes(64L) + .withRecordCount(1L) + .withReferencedDataFile("s3://b/db1/" + referencedDataFile) + .build()) + .commit(); + } + + /** Commits an equality-delete file as a new snapshot (a distinct delete manifest). */ + static void addEqualityDeleteSnapshot(InMemoryCatalog catalog, String table, String fileName) { + catalog.loadTable(id(table)).newRowDelta() + .addDeletes(FileMetadata.deleteFileBuilder(PartitionSpec.unpartitioned()) + .ofEqualityDeletes(1) + .withPath("s3://b/db1/" + fileName) + .withFormat(FileFormat.PARQUET) + .withFileSizeInBytes(64L) + .withRecordCount(1L) + .build()) + .commit(); + } + static ConnectorSession session(String timeZone) { return new FakeSession(timeZone); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsActionTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsActionTest.java index 29604f0f13a756..ef81cf3fed474e 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsActionTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/action/IcebergExpireSnapshotsActionTest.java @@ -27,14 +27,20 @@ import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Iterables; +import org.apache.iceberg.FileContent; +import org.apache.iceberg.ManifestFile; +import org.apache.iceberg.Snapshot; +import org.apache.iceberg.Table; import org.apache.iceberg.catalog.TableIdentifier; import org.apache.iceberg.inmemory.InMemoryCatalog; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import java.util.Collections; +import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.Set; /** * Pins {@code expire_snapshots}: the validation pass and the six-counter result. @@ -117,6 +123,52 @@ public void resultSchemaIsSixBigintCounters() { } } + @Test + public void buildDeleteFileContentMapDedupsManifestsSharedAcrossSnapshots() { + // WHY: expire_snapshots builds the delete-file -> content classification map by scanning every snapshot's + // delete manifests. Iceberg carries immutable delete manifests forward across snapshots, so the naive + // per-snapshot scan re-reads the same manifest once per referencing snapshot. Deduping by manifest path + // must (a) read each DISTINCT delete manifest exactly once — strictly fewer than the per-snapshot total — + // and (b) leave the resulting map byte-identical (every delete path still classified correctly). A + // regression that reset the visited set per snapshot (no dedup) or skipped a distinct manifest (dropping + // its delete files) is killed here. + InMemoryCatalog catalog = ActionTestTables.freshCatalog(); + TableIdentifier id = ActionTestTables.id("t"); + ActionTestTables.createTable(catalog, "t"); + ActionTestTables.appendSnapshot(catalog, "t", "f1.parquet", 1L); + // A position-delete snapshot creates delete manifest DM1... + ActionTestTables.addPositionDeleteSnapshot(catalog, "t", "pos-del.parquet", "f1.parquet"); + // ...an equality-delete snapshot creates DM2 and carries DM1 forward... + ActionTestTables.addEqualityDeleteSnapshot(catalog, "t", "eq-del.parquet"); + // ...and two more appends carry both delete manifests forward unchanged (the redundant re-read source). + ActionTestTables.appendSnapshot(catalog, "t", "f2.parquet", 1L); + ActionTestTables.appendSnapshot(catalog, "t", "f3.parquet", 1L); + + Table table = catalog.loadTable(id); + Set distinctDeleteManifests = new HashSet<>(); + int perSnapshotReads = 0; + for (Snapshot snapshot : table.snapshots()) { + for (ManifestFile manifest : snapshot.deleteManifests(table.io())) { + distinctDeleteManifests.add(manifest.path()); + perSnapshotReads++; + } + } + // Sanity: the fixture must actually reuse a delete manifest across snapshots, else the test proves nothing. + Assertions.assertTrue(perSnapshotReads > distinctDeleteManifests.size(), + "fixture must share a delete manifest across snapshots: perSnapshot=" + perSnapshotReads + + " distinct=" + distinctDeleteManifests.size()); + + IcebergExpireSnapshotsAction action = action(ImmutableMap.of("retain_last", "1")); + Map contentByPath = action.buildDeleteFileContentMap(table); + + Assertions.assertEquals(distinctDeleteManifests.size(), action.lastDeleteManifestReadCount, + "each distinct delete manifest must be read exactly once (dedup), not once per referencing snapshot"); + Assertions.assertEquals(FileContent.POSITION_DELETES, contentByPath.get("s3://b/db1/pos-del.parquet"), + "the position-delete file must stay classified as POSITION_DELETES"); + Assertions.assertEquals(FileContent.EQUALITY_DELETES, contentByPath.get("s3://b/db1/eq-del.parquet"), + "the equality-delete file must stay classified as EQUALITY_DELETES"); + } + @Test public void rejectsNegativeOlderThanTimestamp() { DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, From e42da633e9e9a14e37e5d2cd754056ebdd6853e7 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 22:26:31 +0800 Subject: [PATCH 22/47] [doc](catalog) fe-connector-iceberg PERF-08: design + summary + tracker updates; next = P2 remaining Records the maintenance-path fixes (rewrite_data_files union registration C19, expire_snapshots delete-manifest dedup C21): the design + red-team verdict, the summary with both commit hashes, tasklist marked done, a progress entry, and the HANDOFF rewritten to point at the remaining P2 tasks (PERF-09/10/11). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 63 ++++----- ...-PERF-08-maintenance-path-rescan-design.md | 124 ++++++++++++++++++ ...PERF-08-maintenance-path-rescan-summary.md | 32 +++++ plan-doc/perf-hotpath-iceberg/progress.md | 14 ++ plan-doc/perf-hotpath-iceberg/tasklist.md | 11 +- 5 files changed, 209 insertions(+), 35 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-08-maintenance-path-rescan-design.md create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-08-maintenance-path-rescan-summary.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index 76a223189fffdc..c623a712ede473 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -6,54 +6,57 @@ --- -# ✅ 已完成（PERF-01 ~ 07） +# ✅ 已完成（PERF-01 ~ 08） -- **PERF-01**（`484f0e0c125`）：胖 handle + 跨查询 `IcebergTableCache`（**凭证 gate**）。规划期 loadTable 3~7→1。 -- **PERF-02**（`518d0599cbf`）：`IcebergPartitionCache`（键 `(TableIdentifier, snapshotId)`、无 gate）。 -- **PERF-03**（`0b96f2e6c78`）：`IcebergFormatCache`（键 `(TableIdentifier, currentSnapshotId)`、无 gate）。 -- **PERF-04**（`2e5f393779c`）：抽惰性 `cacheBackedFileScanTasks` 三处复用，manifest cache 接回大表流式 + COUNT(*)。 -- **PERF-05**（`aea3ebdd40e`）：`IcebergCommentCache`（键 `TableIdentifier`、无 gate），**仅 vended 非 session=user**。 -- **PERF-06**（`6294edf2833`）：vended-credentials URI 归一化 scan 级「准备一次、逐文件套用」normalizer。 -- **PERF-07**（`97bdcd6bdbe` fe-core + `ea7fd1f6e7a` iceberg）：每语句 `ConnectorStatementScope`（挂 `StatementContext`、构造期捕获）作读写共享的表加载归属者，一条语句一张表只加载一次；拆胖句柄；delete stash 下沉到作用域（删单例 + `IcebergRewritableDeleteStash` 类 + 测试）；v3 行级 DML + NONE fail-loud。写侧收益≈0，交付=架构连贯。**净增 fe-core SPI（碰铁律 A，用户签字）**。设计+小结见 `designs/FIX-PERF-07-unified-*`。 -- iceberg 全模块单测绿（**968 pass / 1 skip**），fe-core 两段验 `ConnectorStatementScopeTest`+`ConnectorSessionImplTest` 绿，0 回归。 +- **PERF-01~07**：见 `progress.md` + `tasklist.md`（胖 handle + 跨查询 `IcebergTableCache`；分区视图缓存；`file_format_type` 兜底 memoize；manifest cache 两旁路接回；information_schema comment 缓存；vended-cred 每文件 normalizer；每语句 `ConnectorStatementScope` 读写共享一次加载）。 +- **PERF-08（`89cc39c8d88` C19 fe-core + `be0035eff62` C21 iceberg）**：两条维护命令的「逐单位重扫/无去重」。 + - C19 `rewrite_data_files`：`ConnectorRewriteDriver` STEP3 逐组 `registerRewriteSourceFiles`（每次一遍全表 `planFiles()`）→ `unionSourceFilePaths(groups)` 并集**一次**登记；整表扫描 G+1→1。行为不变靠同一固定快照 + 组互斥 + 连接器按 path 去重。 + - C21 `expire_snapshots`：`buildDeleteFileContentMap` 加方法作用域 visited set 按 `manifest.path()` 去重；远程读 O(S×M)→O(distinct M)、结果 map 逐字节不变。 + - 2 独立对抗 agent 均判 parity 保持、0 破坏性发现。`ConnectorRewriteDriverTest` 5/5 + `IcebergExpireSnapshotsActionTest` 9/9，两处 BUILD SUCCESS + 0 checkstyle。权威设计/小结/红队见 `designs/FIX-PERF-08-maintenance-path-rescan-*`。 --- -# 🆕 下一个 session = **PERF-08（维护路径逐单位重扫 / 无去重，C19+C21）** +# 🆕 下一个 session = **P2 剩余（PERF-09 / PERF-10 / PERF-11 三选一）** -> 改动小、收益明确、**不碰读写热路径**（相较 PERF-07 是轻活）；覆盖两条独立维护命令，同一「逐单位重扫、零去重」模式；实现上可各自独立 commit。**行号信 grep**（PERF-07 后可能又漂）。 +> 剩三项，均 P2（CPU/payload/框架层，触发门槛较高但模式典型）。**开场先按 README 流程 step 4 向用户复述并让其定下一个做哪个**（下面给推荐 + 各自要点；**行号信 grep**，PERF-07/08 后可能又漂）。 -## C19 — `rewrite_data_files`（分布式 compaction 规划） -- **病灶**：`ConnectorRewriteDriver.run` STEP3（约 :143）**每个** rewrite group 各调一次 `registerRewriteSourceFiles` → 每次一遍 `useSnapshot(...).planFiles()`（整表 manifest 扫描）。G 个 group = **G+1 次整表扫描**，G~50-200 时分钟级。 -- **修复方向**：**union 所有 group 一次注册**（SPI `registerRewriteSourceFiles` 本就收 `Set`）——把 G 次调用坍缩为 1 次，扫一遍 planFiles、按 group 分派。 -- **⚠ 落地前 grep 定位**：`ConnectorRewriteDriver`（fe-core 通用驱动？还是连接器侧？）的 STEP3 循环真实结构 + `registerRewriteSourceFiles` 的连接器实现（iceberg 侧 `IcebergConnectorTransaction.registerRewriteSourceFiles`，PERF-07 recon 记其在 `:362`、re-derive 在 `:380-381`、当前 `startingSnapshotId`）；确认「union 一次注册」不破分布式 rewrite 的 per-group 文件归属/OCC。 +## 推荐先做：PERF-10（C8）——WHERE conjunct 每查询转换 5~6 次 +- **轻活、iceberg-local、与已完成的 PERF-01 同一失效点**（node 字段 memo，宜顺手）。 +- **病灶**：同组 conjunct 在 `buildRemainingFilter`(×3) + `buildScan` + `getScanNodeProperties`（EXPLAIN 专用序列化，非 EXPLAIN 也跑）被 `convertPredicate` 转换 5~6 次/查询。单次微秒~毫秒，与簇1 同源叠加。 +- **修复方向**：node 字段 memo，与 `cachedPropertiesResult` **同点失效**（复用 PERF-01 立住的失效收窄模式）。 +- **⚠ 落地前 grep**：`IcebergScanPlanProvider` 里 `convertPredicate` 的 5~6 个调用点真实位置 + 各自输入是否恒等（同一 conjunct 列表）→ 确认可安全 memo（同输入同输出、谓词不变量）。 -## C21 — `expire_snapshots` -- **病灶**：`IcebergExpireSnapshotsAction.buildDeleteFileContentMap`（约 :271-293）对**每个** snapshot 读其全部 delete manifest、**无 visited-path 去重** —— 相邻 snapshot 的 manifest 大量重叠，S×M 串行远程读。 -- **修复方向**：按 `ManifestFile.path()` **去重坍缩为 O(distinct)**（visited set）。 -- **⚠ 落地前 grep 定位**：`IcebergExpireSnapshotsAction` 真实结构 + 该 map 的消费方（确认按 path 去重不丢任何该删的 content 条目）。 +## PERF-09（C5）——streaming pump 逐 split 重建 backend 候选集（**fe-core 框架层**） +- **⚠ 改的是 fe-core 通用框架**（惠及所有连接器、非 source-specific，**允许改**）；但属共享热路径，须证 **byte + cost 对所有连接器双不变**（对齐「共享框架须双不变」纪律 + 保持 connector-agnostic）。 +- **病灶**：`startStreamingSplit:~1638` 逐 split `addToQueue` → `SplitAssignment` 每 split 重建 backend 候选集（`FederationBackendPolicy.computeScanRangeAssignment` 全量 backend 拷贝 + shuffle + multimap）+ `synchronized` 往返。10⁵~10⁶ split × ~100 BE ≈ 10⁷~10⁸ 冗余。 +- **修复方向**：pump 侧微批（64~256/批）。**先 grep** 该泵 + `SplitAssignment` 现结构，确认微批不改分配语义（同一 split 集合的 backend 分配结果不变）。 -## 铁律提醒（PERF-08） -- 维护路径修复**不碰** fe-core 读写热缝、不碰 PERF-07 的作用域；若发现 C19 的驱动在 fe-core 通用层，遵「共享框架须 byte+cost 双不变」纪律 + 保持 connector-agnostic（对齐 `catalog-spi-plugindriven-no-source-specific-code`）。 -- 立项先读审计报告 §5 对 C19/C21 的原始证据，再对照 HEAD 真实代码 review（HANDOFF 行号/方案可能过时）。 +## PERF-11（簇5，C12 C13 C14 C15）——per-split 不变量 + payload 放大（**批量、可拆多 commit**） +- 同一 `buildRange`/`populateRangeParams` 区域：C12 每 byte-slice 重算 partition JSON/identity map/delete 转换 →(specId,PartitionData) memo；C13 v3 delete 双重 thrift 转换 → 转一次 per-file 复用；C14 通用节点 per-split `LocationPath.of`/造完即弃的 columns-from-path/`getFileCompressType` → hoist（**⚠ 涉 fe-core 通用节点，保持 connector-agnostic**）；C15 delete 列表+partition JSON 逐 slice 复制进 `TFileRangeDesc` → per-file 共享引用。 +- **先 grep** `buildRange` / `populateRangeParams` / `IcebergScanRange` 真实结构再逐条立项。 + +## 铁律提醒 +- PERF-10/11(iceberg 部分) 修复放**连接器侧** node 字段/memo，不碰 fe-core（对齐 PERF-01 模式）。 +- PERF-09 与 PERF-11 的 C14 碰 **fe-core 通用节点/框架**：允许改（跨连接器普惠）但**禁按源名分支** + 须证 byte+cost 双不变。 +- 立项先读审计报告 §2/§5 对应簇原始证据（findings.json 有完整调用链），再对照 HEAD 真实代码 review（HANDOFF 行号/方案可能过时）。 --- # 🧰 构建/验证坑（**实证，务必照做**） -1. **iceberg 侧测试**：`mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='<类逗号列表>' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`（绝对 `-f /fe/pom.xml`）。`${revision}` 未 flatten→必 `-am`；`-am test` 不产 shade jar 故用 **`install`**；类列表存 `scratchpad/iceberg-tests.txt`（`ls src/test/.../*Test.java | xargs -n1 basename | sed 's/.java//' | paste -sd,`）。 -2. **动 fe-core 者必两段验**：iceberg 连接器不依赖 fe-core，`-pl iceberg -am` 反应堆无 fe-core——fe-core 改动 + 其单测须另跑 `mvn test -pl fe-core -am -Dtest= -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -f /fe/pom.xml`。（PERF-08 若纯 iceberg 则免；若碰 fe-core 驱动则要。） +1. **iceberg 侧测试**：`mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='<类逗号列表>' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`（绝对 `-f /fe/pom.xml`）。`${revision}` 未 flatten→必 `-am`；`-am test` 不产 shade jar 故用 **`install`**。 +2. **动 fe-core 者必两段验**：iceberg 连接器不依赖 fe-core，`-pl iceberg -am` 反应堆无 fe-core——fe-core 改动 + 其单测须另跑 `mvn test -pl fe-core -am -Dtest= -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -f /fe/pom.xml`。（**PERF-09 与 PERF-11 的 C14 碰 fe-core → 要两段验；纯 iceberg 项免**。）PERF-08 已实证：C19 走 fe-core 段、C21 走 iceberg 段。 3. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 静默跳过）。 -4. **别用 `mvn -q` 跑测试**：抑制 `BUILD SUCCESS`/`Tests run`。用 surefire XML 或 grep `Tests run:`。 +4. **别用 `mvn -q` 跑测试**：抑制 `BUILD SUCCESS`/`Tests run`。grep 日志 `BUILD SUCCESS` + `Tests run:`。 5. **别 `nohup ... &` 套 `run_in_background`**——maven 变孤儿、假 exit 0。直接 `mvn ... >> log 2>&1`（run_in_background）；**通知里 exit code 是 echo 的**，grep 日志 `BUILD SUCCESS`。 -6. **checkstyle 主源 ≤120**，且**扫 test 源**（PERF-07 踩坑：删测试后遗留 unused import 挂 checkstyle）——删测试后 grep 残留 import。import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行。 -7. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add（`git rm` 会自动 stage 删除，拆 commit 时用 `git restore --staged` 剔出）。 +6. **checkstyle 主源 ≤120**，且**扫 test 源**（删测试/改 import 后 grep 残留 unused import）。import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行。PERF-08 踩过：test helper 内联 builder 不具名类型 → 该 import 变 unused。 +7. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。工作树另有大量**与本任务无关的 untracked 杂物**（`.claude/`、`plan-doc/` 其它、`docker/`…），提交只 add 本任务具体文件。 8. **并发探测**：本目录 `find fe -newermt '-120 sec'` + `pgrep -a -f maven`。`/mnt/disk1/yy/git/doris` 是另一 worktree，不冲突。 9. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 --- -# 🗂 PERF-07 遗留（已闭环，仅记录） +# 🗂 遗留（已闭环，仅记录） -- e2e 回归（独立 + HMS 网关目录 INSERT/DELETE/MERGE/OVERWRITE 无复活 + 分布式 rewrite + 跨 catalog MERGE + session=user/vended 归一）**留 P6.6 切换阶段统一补**（对齐 `hms-iceberg-delegation-needs-e2e`；当前 iceberg 未在 `SPI_READY_TYPES`，provider 休眠）。 -- 架构记忆 `iceberg-table-resolution-cache-scoping` 待按本次落地更新（原记「stash 下单例待用户定高度」已由完整统一版闭环）。 +- **e2e 统一补**：PERF-07 的读写归一 + PERF-08 的 `rewrite_data_files`/`expire_snapshots` 计数断言 e2e，均**留 P6.6 切换阶段统一补**（当前 iceberg 未在 `SPI_READY_TYPES`，provider 休眠；对齐 `hms-iceberg-delegation-needs-e2e`）。PERF-08 的 C19 分布式实路径单测非集群不可达，亦归此。 +- 架构记忆 `iceberg-table-resolution-cache-scoping` 已随 PERF-07 落地闭环。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-08-maintenance-path-rescan-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-08-maintenance-path-rescan-design.md new file mode 100644 index 00000000000000..285589777de24c --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-08-maintenance-path-rescan-design.md @@ -0,0 +1,124 @@ +# FIX-PERF-08 — 维护路径逐单位重扫 / 无去重（C19 + C21）设计 + +> 一个任务、两处独立修复（各自独立 commit）。同一「逐单位重扫、零去重」模式。 +> 硬约束：**性能修复 → 行为不变**，只减远程读，不改任何可观测结果 / 异常 / 提交后表状态。 +> 基线：分支 `catalog-spi-review-16`，行号已按 HEAD 重新 grep（PERF-07 后）。 + +--- + +## Problem + +两条 `ALTER TABLE ... EXECUTE` 维护命令，各自把「读一次就够」的整表元数据扫描按「单位数」重复了 N 遍： + +- **C19 `rewrite_data_files`**：分布式 compaction 规划把 G 个 bin-pack 组的 source 文件**逐组注册**，每次注册在连接器侧触发一遍**整表 `planFiles()` 扫描**（读 manifest-list + 全部 manifest）。G 个组 = **G 次整表扫描**；G 常见 50~200，分钟级。 +- **C21 `expire_snapshots`**：过期前构建「delete 文件 → 内容类型」映射时，**逐 snapshot** 读该 snapshot 的**全部** delete manifest，**无 visited 去重**。相邻 snapshot 的 delete manifest 大量重叠（iceberg manifest 不可变、跨 snapshot 原样前推），同一 manifest 被远程读 S 次。S×M 串行远程读。 + +--- + +## Root Cause + +- **C19**：`ConnectorRewriteDriver.run()` STEP3（fe-core）逐组调 SPI `registerRewriteSourceFiles`；iceberg 实现 `IcebergConnectorTransaction.registerRewriteSourceFiles`（:366）每次调用都跑一遍 `table.newScan().useSnapshot(startingSnapshotId).planFiles()` 把 raw path 反解回 `DataFile`。关键事实：**所有逐组调用 pin 的是同一个 `startingSnapshotId`**（一条共享事务、beginWrite 时捕获一次），且**组之间互斥**（bin-pack，每个 data file 只属于一个组）。因此 G 次「同快照、各自过滤子集」的扫描，本可合成一次「同快照、过滤并集」的扫描。SPI 契约本就写明 `registerRewriteSourceFiles` "Accumulates across calls" 且入参是 `Set` —— 一次传并集与 G 次传分片，累积结果完全一致。 +- **C21**：`IcebergExpireSnapshotsAction.buildDeleteFileContentMap`（:271）双层循环 `for snapshot → for deleteManifest → readDeleteManifest`，没有「这个 manifest 我读过了」的 visited 判断。iceberg 的 `ManifestFile` 不可变，读同一 manifest 得到的 `DeleteFile` 集合恒等，且 `putIfAbsent` 对重复 path 本就是 no-op —— 逐 snapshot 重读纯属浪费。 + +--- + +## Design + +### C19 — union 一次注册（改 fe-core 通用驱动，保持 connector-agnostic） + +`ConnectorRewriteDriver.run()` STEP3 由「逐组循环调用」改为「先并集、再一次调用」： + +```java +// STEP 3: register the UNION of every group's source data files in a single call. The connector re-derives +// them from the table at the pinned OCC snapshot with ONE planFiles() scan; the previous per-group loop +// repeated that full-table scan once per group (PERF-08 / C19). Ordering is unchanged — still after +// runGroups (so the table + OCC snapshot are loaded) and before commit. +Set sourceFilePaths = unionSourceFilePaths(groups); +connectorTx.registerRewriteSourceFiles(sourceFilePaths); +``` + +并集抽成一个包私有静态纯函数 `static Set unionSourceFilePaths(List groups)`（唯一目的：让并集/去重逻辑可单测；见 Test Plan）。iceberg 连接器**零改动**。 + +**为什么行为不变**（逐条对齐红队关注点）： +1. 同快照：G 次调用与并集 1 次调用都用同一 `startingSnapshotId`（共享事务，恒定）→ 反解结果同源。 +2. 累积集合一致：`registerRewriteSourceFiles` 内 `matched` 按 path 去重；组之间互斥 → `filesToDelete`（`List`）内容与逐组累积**逐元素一致**（仅可能顺序不同，`RewriteFiles.deleteFile` 是集合语义，顺序无关）。 +3. fail-loud 一致：任一 path 在 pin 快照缺失，逐组版在第 k 组抛、并集版在唯一一次调用抛，均 `DorisConnectorException`；STEP2 的 `catch` 都会 `rollback` 整条事务，部分注册无副作用。 +4. 结果行不受影响：STEP5 的四列计数来自 `ConnectorRewriteGroup` 元数据（组级 `getDataFileCount/getTotalSizeBytes`）与 commit 后的 added-count，**不读** `filesToDelete.size()`。 +5. 空输入：空组不贡献并集；全空 → 并集空 → `registerRewriteSourceFiles` 早退（与逐组全空等价）。 + +**约束核对**：`ConnectorRewriteDriver` 在 fe-core，但本改动 (a) 不引入 source-specific 分支（仍只搬运 `Set` 中性路径，无 iceberg 类型）；(b) 不把连接器逻辑/属性解析/缓存搬进 fe-core —— 只是把**已有** driver 对**已有** SPI 的调用方式从 G 次收敛为 1 次，是行为收敛而非能力上移。符合「fe-core 只出不进 + 禁 scaffolding 搬迁」两律与 connector-agnostic 纪律。 + +### C21 — 按 manifest path 去重（连接器侧，纯 iceberg 局部） + +`buildDeleteFileContentMap` 内层加 visited set： + +```java +Set visitedDeleteManifests = new HashSet<>(); +for (Snapshot snapshot : icebergTable.snapshots()) { + List deleteManifests = snapshot.deleteManifests(icebergTable.io()); + if (deleteManifests == null || deleteManifests.isEmpty()) { + continue; + } + for (ManifestFile manifest : deleteManifests) { + if (!visitedDeleteManifests.add(manifest.path())) { + // Immutable manifests are carried across adjacent snapshots unchanged; reading one again yields + // the identical DeleteFile set (putIfAbsent already made the re-read a no-op). Skip the remote read. + continue; + } + try (CloseableIterable deleteFiles = ManifestFiles.readDeleteManifest( + manifest, icebergTable.io(), icebergTable.specs())) { + for (DeleteFile deleteFile : deleteFiles) { + deleteFileContentByPath.putIfAbsent(deleteFile.location(), deleteFile.content()); + } + } + } +} +``` + +**为什么行为不变**：manifest 不可变 → 读每个 distinct manifest 一次即覆盖当前代码看到的全部 delete 文件（当前代码读的是同一批 distinct manifest 的带重复超集）；`putIfAbsent` first-win 因 `deleteFile.content()` 对给定 location 恒定而与遍历顺序无关 → **结果 map 逐字节一致**，只是远程读从 O(S×M) 降到 O(distinct M)。 + +**度量守门**：新增 `@VisibleForTesting` 计数器记录本次 `readDeleteManifest` 实际调用次数，测试断言「S 个 snapshot 共享同一 delete manifest 时，读次数 == distinct manifest 数」而非 S×M。 + +--- + +## Implementation Plan + +两个独立 commit： + +1. **commit A（C19，fe-core）**：`ConnectorRewriteDriver` STEP3 循环 → 并集一次调用 + 抽 `unionSourceFilePaths` 静态纯函数；新增 `ConnectorRewriteDriverTest` 用例断言并集/去重/空组。**须两段验**（动了 fe-core：iceberg 反应堆不含 fe-core）。 +2. **commit B（C21，iceberg）**：`buildDeleteFileContentMap` 加 visited 去重 + `@VisibleForTesting` 读计数；`buildDeleteFileContentMap` 降为包私有以便直测；`ActionTestTables` 加 position/equality delete 播种 helper；`IcebergExpireSnapshotsActionTest` 加去重 + 分类 parity 用例。**纯 iceberg，免 fe-core 段**。 + +--- + +## Risk Analysis + +- **C19 顺序依赖**：STEP3 必须在 `runGroups` 之后（table + snapshot 已加载）、commit 之前。并集调用位置不变，只改循环体，风险低。 +- **C19 未来非 iceberg 连接器**：SPI 契约明确「累积 + 收 Set」，一次传并集在契约内；且当前仅 iceberg override。低。 +- **C19 单测覆盖有限**：driver 的分布式实路径（非空组 → 真 BE INSERT-SELECT）在类注释里就明确「留 flip rehearsal e2e」，STEP3 非集群不可达。故 C19 的单测只能守到「并集函数正确」这一层；「一次调用」的端到端由既有连接器侧 `registerRewriteSourceFilesThenCommit...` 测试（一次调用注册多 path）+ 后续 flip rehearsal e2e 兜底。诚实标注，不假装守全。 +- **C21 destructive**：`execute()` 真过期快照；单测直调包私有 `buildDeleteFileContentMap`（非 destructive）验证 map + 读计数，避开提交副作用。 +- **C21 delete 文件跨多个 distinct manifest**：两 manifest 都会被读（都是 distinct path），`putIfAbsent` first-win 因 content 恒定而顺序无关 → 一致。低。 + +## Test Plan + +### Unit Tests +- **C19**（`ConnectorRewriteDriverTest`）：`unionSourceFilePaths` 对多组返回去重并集；跨组重复 path 只留一份；含空组时空组无贡献；空列表 → 空集。 +- **C21**（`IcebergExpireSnapshotsActionTest` + `ActionTestTables`）： + - 去重守门：构造 delete manifest 被 S 个 snapshot 前推共享的表，断言 `readDeleteManifest` 实际读次数 == distinct manifest 数（< S×M）。 + - 分类 parity：position + equality delete 文件分布在重叠 snapshot 上，断言结果 map 对每个 delete path 的 `FileContent` 分类正确（catch 去重误跳含唯一 delete 的 manifest 的回归）。 + +### E2E Tests +- 维护命令 e2e（独立 iceberg 目录 + HMS 网关委派跑 `rewrite_data_files` / `expire_snapshots` 断言计数）随 P6.6 切换阶段统一补（当前 iceberg 未进 `SPI_READY_TYPES`，provider 休眠；对齐 `hms-iceberg-delegation-needs-e2e`）。 + +--- + +## 设计红队（对抗 parity review，2 独立 agent） + +两条修复均判 `parity_preserved = true`，无 breaks-parity 发现。采纳的残余提醒： + +- **C19（medium，非破坏）**：union 的 List 去重正确性依赖「组之间 path 互斥」——iceberg 侧可证（`planFiles()` 每 data file 一个 task + bin-pack 分桶划分），但 SPI 未强制。**落地动作**：STEP3 注释显式记该不变量；补空组 no-op parity 单测。stale-plan 报错文案的计数从「首个缺失组」变为「并集」——同类型异常、同 rollback/不提交结局，非稳定契约，可接受。 +- **C21（medium，非破坏但可能使优化空转）**：`visited` set **必须在 snapshot 外层循环之前（方法作用域）声明**——若误放进内层则每轮重置 → dedup 空转（仍 parity，但零收益）。**落地动作**：确保方法作用域声明；单测用「同一 delete manifest 被 S 个 snapshot 前推共享」断言读次数坍缩。transient IO 失败面变小（读得少 → 失败机会少）属预期收益，非破坏。 + +## 度量收益 + +- **C19**：整表 `planFiles()` 扫描 G+1 → **1**（G≈50~200 时分钟级 → 秒级）。 +- **C21**：delete manifest 远程读 O(S×M) → **O(distinct M)**（相邻 snapshot 重叠越多，收益越大）。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-08-maintenance-path-rescan-summary.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-08-maintenance-path-rescan-summary.md new file mode 100644 index 00000000000000..63ebb9bfeb6c8b --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-08-maintenance-path-rescan-summary.md @@ -0,0 +1,32 @@ +# FIX-PERF-08 — 维护路径逐单位重扫 / 无去重（C19 + C21）小结 + +> 设计 + 红队详见 [`FIX-PERF-08-maintenance-path-rescan-design.md`](./FIX-PERF-08-maintenance-path-rescan-design.md)。 +> 一个任务、两处独立修复、两个独立 commit。硬约束达成：只减远程读，行为逐字节不变。 + +## Problem + +两条 iceberg `ALTER TABLE ... EXECUTE` 维护命令把「读一次就够」的整表元数据扫描按单位数重复执行： +- `rewrite_data_files`：G 个 rewrite group **逐组**登记源文件 → 连接器每次一遍整表 `planFiles()` → **G+1 次整表扫描**。 +- `expire_snapshots`：构建 delete 分类表时**逐 snapshot** 读全部 delete manifest，**无去重** → 相邻 snapshot 大量重叠 → **S×M 串行远程读**。 + +## Root Cause + +- C19：`ConnectorRewriteDriver.run()` STEP3 逐组调 `registerRewriteSourceFiles`；iceberg 实现每次跑一遍 `useSnapshot(startingSnapshotId).planFiles()`。所有调用 pin 同一快照、组间互斥 → 可合成一次并集扫描。 +- C21：`IcebergExpireSnapshotsAction.buildDeleteFileContentMap` 双层循环无 visited 判断；iceberg manifest 不可变、跨 snapshot 原样前推 → 同一 manifest 被重复远程读。 + +## Fix + +- **C19（fe-core，commit `89cc39c8d88`）**：STEP3 逐组循环 → `unionSourceFilePaths(groups)` 并集**一次登记**。抽包私有静态纯函数便于单测。iceberg 连接器零改动。行为不变靠：同一固定快照 + 组互斥 + 连接器按 path 去重 → `filesToDelete` 逐元素一致；四列计数来自组元数据；缺失路径两种写法都 fail-loud + rollback。 +- **C21（iceberg，commit `be0035eff62`）**：`buildDeleteFileContentMap` 加**方法作用域** visited set，`if (!visitedDeleteManifests.add(manifest.path())) continue;` 同一 manifest 只读一次。结果 map 逐字节不变（manifest 不可变 + `putIfAbsent` 与顺序无关）。新增 `@VisibleForTesting int lastDeleteManifestReadCount` 作度量守门。 + +## Tests + +- **C19**（`ConnectorRewriteDriverTest`，+2）：`unionSourceFilePaths` 多组并集、跨组重复去重、含空组、空 plan → 5/5 绿。分布式实路径留 flip rehearsal e2e（既有约定）。 +- **C21**（`IcebergExpireSnapshotsActionTest`，+1；`ActionTestTables` 加 position/equality delete 播种 helper）：构造「同一 delete manifest 被多 snapshot 前推共享」的表，断言实际读次数 == distinct manifest 数（< 逐 snapshot 总数），且 position/equality delete 分类正确。 + +## Result + +- **C19**：整表 `planFiles()` 扫描 G+1 → 1。 +- **C21**：delete manifest 远程读 O(S×M) → O(distinct M)。 +- 两条经 2 独立对抗 agent 逐一证伪均判 `parity_preserved=true`，无破坏性发现。 +- 验证：`ConnectorRewriteDriverTest` 5/5、fe-core BUILD SUCCESS、0 checkstyle；iceberg `IcebergExpireSnapshotsActionTest` 9/9 绿、0 checkstyle。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index d4b9747f8a5634..3c4083969cd2ef 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -116,3 +116,17 @@ - **结果**：iceberg **968 pass / 0 fail / 1 skip**；fe-core 两段验 `ConnectorStatementScopeTest` 3 + `ConnectorSessionImplTest` 17 绿；两处 BUILD SUCCESS + checkstyle 0 违规，0 回归。summary 见 `designs/FIX-PERF-07-unified-per-statement-table-owner-summary.md`。 - **新判据（可复用）**：**「唯一贯穿语句的对象是 `StatementContext`」**——连接器 session 一语句被重建 ~26 次、缓存挂它即死；跨读写共享须向上够到 StatementContext(经中性 SPI `getStatementScope()`)，**构造期捕获**(非实时读)才能让 off-thread 扫描泵够到(它们复用请求线程建的同一 session、无 ConnectContext thread-local)。 - **下一步**：见 HANDOFF —— PERF-08（C19/C21 维护路径逐单位重扫/无去重；改动小可先行）。 + +--- + +## 2026-07-18 — session 3：PERF-08 实现 + 全绿（commit `89cc39c8d88` C19 + `be0035eff62` C21） + +- **开场复核（动码前，grep HEAD 校行号）**：PERF-07 后行号有漂——C19 driver STEP3 循环实在 `ConnectorRewriteDriver.java:143-145`（fe-core），iceberg 侧 `IcebergConnectorTransaction.registerRewriteSourceFiles:366`（每次 `useSnapshot(startingSnapshotId).planFiles()` 全表扫）；C21 `buildDeleteFileContentMap:271-293`。关键事实核实：**逐组注册的所有调用 pin 同一 `startingSnapshotId`**（beginWrite 捕获一次、共享事务恒定）、bin-pack 组**互斥**、四列结果计数来自组元数据非 `filesToDelete`；C21 map 只喂 `deleteWith` 分类回调。 +- **设计红队（2 独立对抗 agent，clean-room）**：两条均判 `parity_preserved=true`、0 破坏性发现。采纳提醒：①C19 union 的 List 去重正确性**依赖组互斥**（iceberg 可证，SPI 未强制）——落地在 STEP3 注释显式记该不变量 + 补空组 no-op 单测；stale-plan 报错文案计数从「首个缺失组」变「并集」属 cosmetic 同结局。②C21 `visited` set **必须方法作用域**（放进 snapshot 内层则每轮清零、dedup 空转仍 parity 但零收益）——落地加注释锁定。 +- **实现（两个独立 `[perf]` commit）**： + - **C19 fe-core（`89cc39c8d88`）**：STEP3 逐组循环 → 抽包私有静态纯函数 `unionSourceFilePaths(groups)` 并集**一次**调 `registerRewriteSourceFiles`。iceberg 连接器零改动。铁律核对：仅把已有 driver 对已有中性 SPI（`Set`）的调用从 G 次收敛为 1 次，无 source-specific 分支、无逻辑上移 fe-core → 合「fe-core 只减不增 + connector-agnostic」。 + - **C21 iceberg（`be0035eff62`）**：`buildDeleteFileContentMap` 加方法作用域 `visitedDeleteManifests`，`if (!add(manifest.path())) continue`；降包私有 + `@VisibleForTesting int lastDeleteManifestReadCount` 度量守门。 +- **测试**：C19 `ConnectorRewriteDriverTest` +2（并集去重/空组空 plan）→ 5/5；分布式 STEP3 需真 BE（类注释既定），单测守到并集函数层，端到端留 flip rehearsal e2e。C21 `IcebergExpireSnapshotsActionTest` +1 + `ActionTestTables` 加 position/equality delete 播种 helper；构造「同一 delete manifest 被多 snapshot 前推共享」表，自检 sanity（perSnapshotReads>distinct，防空测）+ 断言读次数==distinct + position/equality 分类 parity → 9/9。 +- **踩坑**：`ActionTestTables` 起初误加未命名的 `DeleteFile` import（builder 内联进 `addDeletes` 不具名）→ checkstyle UnusedImports；删 import + 删一个凑数的 anchor 方法（违简单律）。 +- **结果**：fe-core BUILD SUCCESS + `ConnectorRewriteDriverTest` 5/5 + 0 checkstyle；iceberg BUILD SUCCESS + `IcebergExpireSnapshotsActionTest` 9/9 + 0 checkstyle。度量：C19 整表 `planFiles()` G+1→1；C21 delete manifest 远程读 O(S×M)→O(distinct M)。 +- **下一步**：见 HANDOFF —— P2 剩余（PERF-09 fe-core streaming pump 微批 / PERF-10 WHERE conjunct memo / PERF-11 簇5 per-split 不变量+payload）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index 6b004c1e1bbc92..28e492a8b46c8d 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -22,7 +22,7 @@ | PERF-05 | P1 | C9 | information_schema.tables 每表 loadTable 取 comment → 复核缩小(普通目录 PERF-01 已覆盖);补**无 gate** `IcebergCommentCache` **仅 vended 非 session=user**(红队:session=user 缓存绕过 per-user 授权=泄漏) | — | ✅ 完成 | `aea3ebdd40e` | | PERF-06 | P1 | C3 | REST vended-cred 每 data/delete file 重建 StorageProperties+Configuration → scan 级「准备一次、逐文件套用」normalizer(SPI seam+fe-core override,连接器侧 hoist);~~fe-core 框架 memo~~否决(per-catalog 跨查询共享→并发冲刷退化+凭证保留贴红线) | — | ✅ 完成 | `6294edf2833` | | PERF-07 | P1 | C20 | 一条 DML 重复 load 同表 → 每语句「表加载归属者」(`ConnectorStatementScope` 挂 StatementContext) 读写共享一次加载；拆胖句柄；delete stash 下沉到每语句作用域(删单例+类)；beginWrite 取共享表 | — | ✅ 完成 | `97bdcd6bdbe`+`ea7fd1f6e7a` | -| PERF-08 | P2 | C19 C21 | 维护路径逐单位重扫 / 无去重（rewrite_data_files 每 group planFiles；expire_snapshots S×M）→ union 一次注册 / 按 path 去重 | 改动小可先行 | ⏳ | | +| PERF-08 | P2 | C19 C21 | 维护路径逐单位重扫 / 无去重（rewrite_data_files 每 group planFiles；expire_snapshots S×M）→ union 一次注册 / 按 path 去重 | 改动小可先行 | ✅ 完成 | `89cc39c8d88`(C19)+`be0035eff62`(C21) | | PERF-09 | P2 | C5 | **（fe-core 框架层）** streaming pump 逐 split 重建 backend 候选集 + 锁往返 → 微批 | ⚠ 跨连接器，见约束 | ⏳ | | | PERF-10 | P2 | C8 | 同组 WHERE conjunct 被转换 5~6 次/查询 → node 字段 memo（与 01 同点失效） | 与 01 同源 | ⏳ | | | PERF-11 | P2 | C12 C13 C14 C15 | per-split 不变量重算 + payload 逐 slice 复制 → hoist / (specId,PartitionData) memo / per-file 共享 | 同区域批量 · ⚠ 含 fe-core 通用节点 | ⏳ | | @@ -83,10 +83,11 @@ ## P2 — CPU/payload/维护路径，影响门槛高但模式典型 -### [ ] PERF-08 — 维护路径逐单位重扫/无去重（C19 C21） ·改动小、可先行插队 -> 覆盖两条独立维护命令，同一"逐单位重扫、零去重"模式。同一任务，实现上可各自独立 commit。 -- **C19 rewrite_data_files**：`ConnectorRewriteDriver.run STEP3:143` 每个 rewrite group 调一次 `registerRewriteSourceFiles` → 每次一遍 `useSnapshot(...).planFiles():379-383`。G 个 group = G+1 次整表扫描，G~50-200 时分钟级。修：**union 所有 group 一次注册**（SPI 本来就收 `Set`）。 -- **C21 expire_snapshots**：`IcebergExpireSnapshotsAction.buildDeleteFileContentMap:271-293` 对**每个** snapshot 读其全部 delete manifest、无 visited-path 去重 —— 相邻 snapshot manifest 大量重叠，S×M 串行远程读。修：按 `ManifestFile.path()` 去重坍缩为 O(distinct)。 +### [x] PERF-08 — 维护路径逐单位重扫/无去重（C19 C21） · ✅ `89cc39c8d88`(C19)+`be0035eff62`(C21) +> 覆盖两条独立维护命令，同一"逐单位重扫、零去重"模式。两个独立 commit。权威设计/小结/红队见 [`designs/FIX-PERF-08-maintenance-path-rescan-design.md`](./designs/FIX-PERF-08-maintenance-path-rescan-design.md) + [`-summary.md`](./designs/FIX-PERF-08-maintenance-path-rescan-summary.md)。 +- **C19 rewrite_data_files**（fe-core `89cc39c8d88`）：`ConnectorRewriteDriver.run` STEP3 逐组调 `registerRewriteSourceFiles`（iceberg 每次一遍 `useSnapshot(startingSnapshotId).planFiles()`）→ G+1 次整表扫描。**落地=`unionSourceFilePaths(groups)` 并集一次登记**（同一固定快照 + 组互斥 + 连接器按 path 去重 ⇒ `filesToDelete` 逐元素一致；四列计数来自组元数据）。整表扫描 G+1→1。gate=`ConnectorRewriteDriverTest` 并集/去重/空组单测（分布式实路径留 flip rehearsal e2e）。 +- **C21 expire_snapshots**（iceberg `be0035eff62`）：`buildDeleteFileContentMap` 逐 snapshot 读全部 delete manifest、无去重。**落地=方法作用域 visited set 按 `manifest.path()` 去重**（manifest 不可变 + `putIfAbsent` 顺序无关 ⇒ 结果 map 逐字节不变）。远程读 O(S×M)→O(distinct M)。gate=`@VisibleForTesting lastDeleteManifestReadCount` + 共享 manifest 场景单测。 +- **红队**：2 独立对抗 agent 逐一证伪均 `parity_preserved=true`、0 破坏性发现。iceberg `IcebergExpireSnapshotsActionTest` 9/9 + fe-core `ConnectorRewriteDriverTest` 5/5，两处 BUILD SUCCESS + 0 checkstyle。 ### [ ] PERF-09 — C5：streaming pump 逐 split 重建 backend 候选集（**fe-core 框架层**） - **病灶**：`startStreamingSplit:1638-1642` 逐 split `addToQueue` → `SplitAssignment` 每 split 重建 backend 候选集（`FederationBackendPolicy.computeScanRangeAssignment:225-235` 全量 backend 拷贝 + shuffle + multimap）+ `synchronized` 往返。10⁵~10⁶ split × ~100 BE ≈ 10⁷~10⁸ 冗余操作。 From 1a746ee251b9e7099a074e1fa7def50ebfa219d4 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 23:35:54 +0800 Subject: [PATCH 23/47] [perf](catalog) fe-connector-iceberg: memoize per-file scan-range invariants across a file's byte-slices (PERF-11 / C12+C15a+C13) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause: TableScanUtil.splitFiles cuts one data file into k byte-slice FileScanTasks, and buildRange ran per slice, recomputing values that are identical for the whole file every time: partition_data_json (date/time format + Jackson), the ordered identity partitionValues, and the merge-on-read delete carriers. On a partitioned MOR table at 100k splits that is ~0.5-2s of extra planning CPU plus large garbage; the k ranges also each held their own copies (FE heap), and a v3 scan re-put an identical rewritable-delete supply per slice. Fix: thread a 1-entry per-file cache (PerFileScratch) through buildRangeForTask, the shared choke point of the eager planScanInternal loop and the streaming IcebergStreamingSplitSource. computePerFileInvariants runs once per file (keyed by the shared DataFile instance, which all k slices return from task.file()); the file's remaining slices reuse the cached values, and the k ranges share the same immutable partitionValues Map and deleteCarriers List. Only start/length and the size-proportional selfSplitWeight stay per-slice. The v3 rewritableDeleteSupply put is moved to once per file, on the file's FIRST slice keyed to the new file (covering the last file in the stream — a flush-on-evict would drop it and silently resurrect deleted rows). The streaming scratch is a per-source (= per-scan) field, single-threaded, so it stays O(1) memory (streaming OOM safety preserved). count-pushdown passes a null scratch (single range, no caching). Byte-identical: every cached value is a deterministic function of task.file() and scan-invariant inputs; distinct logical files never share a DataFile instance (the SDK and manifest-cache enumeration both copy per entry), so the identity key never returns a stale value; splitFiles emits a file's slices consecutively, so the 1-entry cache hits 100% (a hypothetical non-consecutive arrival would only cost perf, never correctness); IcebergScanRange is immutable and every consumer reads only. Verified by a 3-lens adversarial byte-parity review (0 breaks). Out of scope (separate follow-ups): the wire-side per-range TFileRangeDesc payload duplication needs a thrift delete-file dictionary + BE reader change (protocol evolution, C15b); the fe-core generic-node per-split hoists are a framework-layer change (C14). Gate: @VisibleForTesting perFileInvariantComputeCount + two tests (a split file computes its invariants once, not per slice; two split files compute twice with no cross-file staleness). IcebergScanPlanProviderTest 105/105, IcebergConnectorTransactionTest 66/66, full iceberg module 1064 pass / 1 skip / 0 fail; 0 checkstyle violations. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../iceberg/IcebergScanPlanProvider.java | 154 +++++++++++++----- .../iceberg/IcebergScanPlanProviderTest.java | 72 ++++++++ 2 files changed, 187 insertions(+), 39 deletions(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index 62fe48ff09cd5d..b53465392350e5 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -37,6 +37,7 @@ import org.apache.doris.thrift.TIcebergFileDesc; import org.apache.doris.thrift.TTableFormatFileDesc; +import com.google.common.annotations.VisibleForTesting; import org.apache.iceberg.BaseFileScanTask; import org.apache.iceberg.BaseTable; import org.apache.iceberg.BatchScan; @@ -236,6 +237,12 @@ public class IcebergScanPlanProvider implements ConnectorScanPlanProvider { // system-table, which fe-core never drains), so the value list is appended single-threaded. private final ConcurrentHashMap> scanProfileStash = new ConcurrentHashMap<>(); + // Test-only gate for the PERF-11 per-file memo: how many times computePerFileInvariants actually ran across + // this provider's scans. A file split into k byte-slices must increment it ONCE (not k times), proving the + // per-slice recompute collapsed to per-file. Not reset per scan — a test uses a fresh provider. + @VisibleForTesting + int perFileInvariantComputeCount; + public IcebergScanPlanProvider(Map properties, IcebergCatalogOps catalogOps) { this(properties, catalogOps, null, null); } @@ -537,6 +544,9 @@ private final class IcebergStreamingSplitSource implements ConnectorSplitSource private CloseableIterator iterator; // Look-ahead buffer so hasNext() can skip data files filtered out by the rewrite scope. private IcebergScanRange buffered; + // Per-file invariant cache (PERF-11): per split-source = per scan; the pump is single-threaded, so the + // 1-entry cache stays O(1) memory (never accumulates), preserving the streaming path's OOM safety. + private final PerFileScratch scratch = new PerFileScratch(); IcebergStreamingSplitSource(CloseableIterable tasks, Table table, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, @@ -562,7 +572,7 @@ public boolean hasNext() { } while (iterator.hasNext()) { IcebergScanRange range = buildRangeForTask(iterator.next(), table, formatVersion, partitioned, - orderedPartitionKeys, zone, uriNormalizer, sliceSize, rewriteScope, null); + orderedPartitionKeys, zone, uriNormalizer, sliceSize, rewriteScope, null, scratch); if (range != null) { buffered = range; return true; @@ -673,6 +683,10 @@ private List planScanInternal( // (buildRange re-attaches task.deletes()), so scoping never drops a delete binding. Set rewriteScope = iceHandle.getRewriteFileScope(); + // Per-file invariant cache (PERF-11): ONE instance per scan, never shared across scans. Reused across + // a data file's consecutive byte-slices so partition JSON / identity map / delete carriers are computed + // once per file, not per slice. The streaming source below holds its own. + PerFileScratch scratch = new PerFileScratch(); List ranges = new ArrayList<>(); try (SplitPlan plan = planFileScanTask(scan, session, table, filter)) { for (FileScanTask task : plan.tasks) { @@ -680,7 +694,7 @@ private List planScanInternal( // identical to the streaming path's IcebergStreamingSplitSource so both produce the same ranges. IcebergScanRange range = buildRangeForTask(task, table, formatVersion, partitioned, orderedPartitionKeys, zone, uriNormalizer, plan.targetSplitSize, rewriteScope, - rewritableDeleteSupply); + rewritableDeleteSupply, scratch); if (range != null) { ranges.add(range); } @@ -739,19 +753,23 @@ private static String rootCauseMessage(Throwable t) { private IcebergScanRange buildRangeForTask(FileScanTask task, Table table, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, UnaryOperator uriNormalizer, long targetSplitSize, Set rewriteScope, - Map> rewritableDeleteSupply) { + Map> rewritableDeleteSupply, PerFileScratch scratch) { DataFile dataFile = task.file(); if (rewriteScope != null && !rewriteScope.contains(dataFile.path().toString())) { return null; } + // First byte-slice of a new data file? (scratch still holds the previous file until buildRange refreshes + // it below — so capture this BEFORE the buildRange call.) The v3 rewritable-delete supply is identical + // for every slice of a file, so record it exactly ONCE per file — on the file's first slice, keyed to + // the new file, so the LAST file in the stream is never dropped. + boolean firstSliceOfFile = scratch.file != dataFile; // targetSplitSize is the scan-level weight denominator (M-2): each data-file range carries a // size-proportional BE scheduling weight (selfSplitWeight computed inside buildRange). IcebergScanRange range = buildRange(table, dataFile, task, formatVersion, partitioned, - orderedPartitionKeys, zone, uriNormalizer, -1, targetSplitSize); - if (rewritableDeleteSupply != null) { + orderedPartitionKeys, zone, uriNormalizer, -1, targetSplitSize, scratch); + if (rewritableDeleteSupply != null && firstSliceOfFile) { // Record this data file's non-equality delete supply keyed on its RAW path (the exact string the BE - // matches a rewritable set against). Splits of one file carry an identical list, so the put is - // idempotent; an empty list (no old non-eq deletes) contributes nothing. + // matches a rewritable set against). An empty list (no old non-eq deletes) contributes nothing. List descs = range.rewritableDeleteDescs(); if (range.getOriginalPath() != null && descs != null && !descs.isEmpty()) { rewritableDeleteSupply.put(range.getOriginalPath(), descs); @@ -1152,7 +1170,7 @@ private List planCountPushdown(Table table, TableScan scan, // targetSplitSize = -1: the count-pushdown collapse emits a single range, so its scheduling // weight is irrelevant → PluginDrivenSplit keeps SplitWeight.standard(). return Collections.singletonList(buildRange(table, task.file(), task, formatVersion, - partitioned, orderedPartitionKeys, zone, uriNormalizer, realCount, -1)); + partitioned, orderedPartitionKeys, zone, uriNormalizer, realCount, -1, null)); } } catch (IOException e) { throw new RuntimeException("Failed to plan iceberg count-pushdown file, error message is:" @@ -1186,16 +1204,92 @@ private CloseableIterable countPushdownFileScanTasks(TableScan sca return scan.planFiles(); } + /** + * Per-file scratch for {@link #buildRange}: the values identical for every byte-slice of one data file + * ({@code TableScanUtil.splitFiles} cuts a file into k slices whose {@code FileScanTask}s all return the + * SAME {@code DataFile} instance from {@code file()}, and emits them consecutively). Computed once on a + * file change and reused across the file's slices, collapsing the per-slice partition-JSON / identity-map / + * delete-carrier recompute to per-file (PERF-11 / C12); the k ranges then share the same immutable + * {@code partitionValues} / {@code deleteCarriers} instances (C15a). The eager loop and the streaming + * source each own ONE instance, never shared across scans. {@code file == null} = fresh / unused. + */ + private static final class PerFileScratch { + private DataFile file; + private Integer partitionSpecId; + private String partitionDataJson; + private Map partitionValues = Collections.emptyMap(); + private List deleteCarriers = Collections.emptyList(); + private String fileFormat; + private Long firstRowId; + private Long lastUpdatedSequenceNumber; + private String rawDataPath; + private String normalizedPath; + } + /** * Build the BE-ready {@link IcebergScanRange} for one {@link FileScanTask}, mirroring legacy * {@code IcebergScanNode.createIcebergSplit} + {@code setIcebergParams}: the file path/offset/size, the * per-file format (native parquet/orc), the table format version, the v3 row-lineage fields, and — for a * partitioned table — the partition spec-id, the all-fields {@code partition_data_json}, and the ordered * identity {@code partitionValues} that become columns-from-path. + * + *

    The per-file-invariant work is memoized through {@code scratch} (keyed by the shared {@code DataFile} + * instance) and reused across the file's byte-slices; only {@code start} / {@code length} / the + * size-proportional {@code selfSplitWeight} are per-slice. A {@code null} scratch (the single + * count-pushdown range) computes without caching. Byte-identical to the former per-slice recompute.

    */ private IcebergScanRange buildRange(Table table, DataFile dataFile, FileScanTask task, int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, - UnaryOperator uriNormalizer, long pushDownRowCount, long targetSplitSize) { + UnaryOperator uriNormalizer, long pushDownRowCount, long targetSplitSize, + PerFileScratch scratch) { + PerFileScratch file = (scratch != null && scratch.file == dataFile) + ? scratch + : computePerFileInvariants(table, dataFile, task, formatVersion, partitioned, + orderedPartitionKeys, zone, uriNormalizer, scratch); + // M-2 size-proportional weight numerator = this split's byte length + the byte size of every + // merge-on-read delete file applying to it, mirroring legacy IcebergSplit.selfSplitWeight (ctor sets + // = length; setDeleteFileFilters adds Σ delete fileSizeInBytes). The delete-size sum is file-invariant + // but the task.length() term is per byte-slice, so the whole weight stays per-slice (NEVER memoized). + // The denominator (targetSplitSize) is passed in; for the single count-pushdown range it is -1 → + // PluginDrivenSplit keeps SplitWeight.standard() (one range, weight irrelevant). + long selfSplitWeight = task.length(); + if (task.deletes() != null) { + for (DeleteFile delete : task.deletes()) { + selfSplitWeight += delete.fileSizeInBytes(); + } + } + return new IcebergScanRange.Builder() + .path(file.normalizedPath) + .originalPath(file.rawDataPath) + .start(task.start()) + .length(task.length()) + .fileSize(dataFile.fileSizeInBytes()) + .fileFormat(file.fileFormat) + .formatVersion(formatVersion) + .partitionSpecId(file.partitionSpecId) + .partitionDataJson(file.partitionDataJson) + .firstRowId(file.firstRowId) + .lastUpdatedSequenceNumber(file.lastUpdatedSequenceNumber) + .partitionValues(file.partitionValues) + .deleteFiles(file.deleteCarriers) + .pushDownRowCount(pushDownRowCount) + .selfSplitWeight(selfSplitWeight) + .targetSplitSize(targetSplitSize) + .build(); + } + + /** + * Compute {@link #buildRange}'s per-file invariants and, when {@code scratch} is non-null, store them into + * it so the file's remaining byte-slices reuse them. Uses {@code task.deletes()} (identical across a + * file's slices) for the delete carriers. Mirrors the former inline per-slice computation exactly — same + * partition-JSON / identity-map ordering, same fail-loud on a non-parquet/orc file, same v3 row-lineage — + * so the memoized range is byte-identical to the per-slice recompute. The scratch fields are assigned only + * after the fail-loud check, so a rejected file never leaves a half-populated scratch. + */ + private PerFileScratch computePerFileInvariants(Table table, DataFile dataFile, FileScanTask task, + int formatVersion, boolean partitioned, List orderedPartitionKeys, ZoneId zone, + UnaryOperator uriNormalizer, PerFileScratch scratch) { + perFileInvariantComputeCount++; Integer partitionSpecId = null; String partitionDataJson = null; Map partitionValues = Collections.emptyMap(); @@ -1236,40 +1330,22 @@ private IcebergScanRange buildRange(Table table, DataFile dataFile, FileScanTask dataFile.fileSequenceNumber() != null && dataFile.firstRowId() != null ? dataFile.fileSequenceNumber() : -1L; } - // M-2 size-proportional weight numerator = this split's byte length + the byte size of every - // merge-on-read delete file applying to it, mirroring legacy IcebergSplit.selfSplitWeight (ctor sets - // = length; setDeleteFileFilters adds Σ delete fileSizeInBytes). task.deletes() is a cached list (no - // extra I/O; buildDeleteFiles reads the same one). The denominator (targetSplitSize) is passed in; for - // the single count-pushdown range it is -1 → PluginDrivenSplit keeps SplitWeight.standard() (one range, - // weight irrelevant), matching the normal-data-only weighting. - long selfSplitWeight = task.length(); - if (task.deletes() != null) { - for (DeleteFile delete : task.deletes()) { - selfSplitWeight += delete.fileSizeInBytes(); - } - } // The range path BE opens is scheme-normalized (legacy createIcebergSplit:852 normalizes via the // 2-arg LocationPath.of(path, storagePropertiesMap)); original_file_path stays raw so BE can match // position-delete entries against the raw iceberg path (legacy setOriginalFilePath:304). String rawDataPath = dataFile.path().toString(); - return new IcebergScanRange.Builder() - .path(uriNormalizer.apply(rawDataPath)) - .originalPath(rawDataPath) - .start(task.start()) - .length(task.length()) - .fileSize(dataFile.fileSizeInBytes()) - .fileFormat(fileFormat) - .formatVersion(formatVersion) - .partitionSpecId(partitionSpecId) - .partitionDataJson(partitionDataJson) - .firstRowId(firstRowId) - .lastUpdatedSequenceNumber(lastUpdatedSequenceNumber) - .partitionValues(partitionValues) - .deleteFiles(buildDeleteFiles(task, uriNormalizer)) - .pushDownRowCount(pushDownRowCount) - .selfSplitWeight(selfSplitWeight) - .targetSplitSize(targetSplitSize) - .build(); + PerFileScratch file = scratch != null ? scratch : new PerFileScratch(); + file.file = dataFile; + file.partitionSpecId = partitionSpecId; + file.partitionDataJson = partitionDataJson; + file.partitionValues = partitionValues; + file.fileFormat = fileFormat; + file.firstRowId = firstRowId; + file.lastUpdatedSequenceNumber = lastUpdatedSequenceNumber; + file.rawDataPath = rawDataPath; + file.normalizedPath = uriNormalizer.apply(rawDataPath); + file.deleteCarriers = buildDeleteFiles(task, uriNormalizer); + return file; } /** diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java index 28abf7eed93f66..91739d5b93773f 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java @@ -382,6 +382,78 @@ public void planScanSplitsLargeFileByFileSplitSizeSessionVar() { Assertions.assertEquals(96 * mb, totalLength, "the split ranges must cover the whole file exactly"); } + @Test + public void planScanMemoizesPerFileInvariantsAcrossByteSlices() { + // PERF-11 (C12/C15a): a data file split into k byte-slices must compute its per-file invariants + // (partition JSON, ordered partition values, delete carriers, format, normalized path) exactly ONCE, + // not once per slice — while each slice keeps its own byte start/length. MUTATION: recomputing per slice + // (dropping the PerFileScratch reuse) -> perFileInvariantComputeCount == k -> red; memoizing start/length + // into the scratch -> the contiguous-tiling assertion -> red. + long mb = 1024L * 1024L; + PartitionSpec spec = PartitionSpec.builderFor(PART_SCHEMA).identity("p").build(); + Table table = createTable("pt", PART_SCHEMA, spec); + table.newAppend() + .appendFile(dataFile(spec, "s3://b/db/pt/p=7/big.parquet", 96 * mb, + Arrays.asList(0L, 32 * mb, 64 * mb), "p=7")) + .commit(); + IcebergScanPlanProvider provider = new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table)); + ConnectorSession session = new FakeScanSession("UTC", + Collections.singletonMap("file_split_size", Long.toString(32 * mb))); + + List ranges = provider.planScan( + session, new IcebergTableHandle("db1", "pt"), Collections.emptyList(), Optional.empty()); + + Assertions.assertTrue(ranges.size() > 1, "expected the 96MB file to split, got " + ranges.size()); + // The per-file invariants were computed ONCE for the whole file — the memo gate. + Assertions.assertEquals(1, provider.perFileInvariantComputeCount, + "per-file invariants must be computed once per file, not once per byte-slice"); + // Every slice carries byte-identical per-file params (partition values + JSON) but its own tiling range. + long expectedStart = 0; + for (ConnectorScanRange r : ranges) { + Assertions.assertEquals(expectedStart, r.getStart(), "slices must tile contiguously from 0"); + Assertions.assertEquals(Collections.singletonMap("p", "7"), r.getPartitionValues(), + "every slice of the file carries the same identity partition values"); + Assertions.assertEquals("[\"7\"]", + populate(r).getTableFormatParams().getIcebergParams().getPartitionDataJson(), + "every slice carries the same partition_data_json"); + expectedStart += r.getLength(); + } + Assertions.assertEquals(96 * mb, expectedStart, "the split ranges must cover the whole file exactly"); + } + + @Test + public void planScanComputesPerFileInvariantsOncePerDistinctFile() { + // The memo recomputes on each file change and never returns a stale file's invariants: two split files + // -> perFileInvariantComputeCount == 2 (once per DISTINCT data file), regardless of total slice count, + // and each file's slices carry that file's OWN partition values. + long mb = 1024L * 1024L; + PartitionSpec spec = PartitionSpec.builderFor(PART_SCHEMA).identity("p").build(); + Table table = createTable("pt", PART_SCHEMA, spec); + table.newAppend() + .appendFile(dataFile(spec, "s3://b/db/pt/p=1/a.parquet", 96 * mb, + Arrays.asList(0L, 32 * mb, 64 * mb), "p=1")) + .appendFile(dataFile(spec, "s3://b/db/pt/p=2/b.parquet", 96 * mb, + Arrays.asList(0L, 32 * mb, 64 * mb), "p=2")) + .commit(); + IcebergScanPlanProvider provider = new IcebergScanPlanProvider(Collections.emptyMap(), opsReturning(table)); + ConnectorSession session = new FakeScanSession("UTC", + Collections.singletonMap("file_split_size", Long.toString(32 * mb))); + + List ranges = provider.planScan( + session, new IcebergTableHandle("db1", "pt"), Collections.emptyList(), Optional.empty()); + + Assertions.assertTrue(ranges.size() >= 4, + "two 96MB files at 32MB splits -> >=4 slices, got " + ranges.size()); + Assertions.assertEquals(2, provider.perFileInvariantComputeCount, + "per-file invariants must be computed once per distinct data file, not per slice"); + Assertions.assertEquals(Collections.singletonMap("p", "1"), + byPath(ranges, "p=1/a.parquet").getPartitionValues(), + "file a's slices carry p=1 (no cross-file staleness)"); + Assertions.assertEquals(Collections.singletonMap("p", "2"), + byPath(ranges, "p=2/b.parquet").getPartitionValues(), + "file b's slices carry p=2 (no cross-file staleness)"); + } + // ── M-2: size-proportional BE scheduling weight (selfSplitWeight / targetSplitSize) ── @Test From 7c5c4b7ee628e1f63258f7de2bcd9bc80d528f01 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 18 Jul 2026 23:39:03 +0800 Subject: [PATCH 24/47] [doc](catalog) fe-connector-iceberg PERF-11 (partial) + PERF-10 deferral: design/summary/tracker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PERF-11 done partially (C12 per-file invariant memo + C15a instance sharing + C13 plan-side put-once, commit 10b7d29423f): design + 3-lens byte-parity red-team verdict + summary; C15b (wire-bytes thrift+BE protocol change) and C14 (fe-core generic-node hoists) recorded as deferred with their prerequisites. PERF-10 (WHERE conjunct conversion) re-verified as low-value and not cleanly fixable within the no-framework-touch constraint; marked deferred with the revival condition. Tasklist rows, progress log, and HANDOFF (next = PERF-09 or PERF-11 remainder) updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 33 +++---- ...-PERF-11-per-file-invariant-memo-design.md | 90 +++++++++++++++++++ ...PERF-11-per-file-invariant-memo-summary.md | 35 ++++++++ plan-doc/perf-hotpath-iceberg/progress.md | 13 +++ plan-doc/perf-hotpath-iceberg/tasklist.md | 22 +++-- 5 files changed, 167 insertions(+), 26 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-per-file-invariant-memo-design.md create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-per-file-invariant-memo-summary.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index c623a712ede473..ef9c83fd705826 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -6,39 +6,32 @@ --- -# ✅ 已完成（PERF-01 ~ 08） +# ✅ 已完成（PERF-01 ~ 08 全，PERF-11 部分） - **PERF-01~07**：见 `progress.md` + `tasklist.md`（胖 handle + 跨查询 `IcebergTableCache`；分区视图缓存；`file_format_type` 兜底 memoize；manifest cache 两旁路接回；information_schema comment 缓存；vended-cred 每文件 normalizer；每语句 `ConnectorStatementScope` 读写共享一次加载）。 -- **PERF-08（`89cc39c8d88` C19 fe-core + `be0035eff62` C21 iceberg）**：两条维护命令的「逐单位重扫/无去重」。 - - C19 `rewrite_data_files`：`ConnectorRewriteDriver` STEP3 逐组 `registerRewriteSourceFiles`（每次一遍全表 `planFiles()`）→ `unionSourceFilePaths(groups)` 并集**一次**登记；整表扫描 G+1→1。行为不变靠同一固定快照 + 组互斥 + 连接器按 path 去重。 - - C21 `expire_snapshots`：`buildDeleteFileContentMap` 加方法作用域 visited set 按 `manifest.path()` 去重；远程读 O(S×M)→O(distinct M)、结果 map 逐字节不变。 - - 2 独立对抗 agent 均判 parity 保持、0 破坏性发现。`ConnectorRewriteDriverTest` 5/5 + `IcebergExpireSnapshotsActionTest` 9/9，两处 BUILD SUCCESS + 0 checkstyle。权威设计/小结/红队见 `designs/FIX-PERF-08-maintenance-path-rescan-*`。 +- **PERF-08（`89cc39c8d88` C19 fe-core + `be0035eff62` C21 iceberg）**：维护命令「逐单位重扫/无去重」——`rewrite_data_files` union 一次登记（全表扫 G+1→1）；`expire_snapshots` 按 `manifest.path()` 去重（O(S×M)→O(distinct M)）。 +- **PERF-10（C8，WHERE conjunct 转换）= 🔬 暂缓**：复核判**低价值**（纯 CPU 微秒、贵的 loadTable 已被 01/07 修掉）且干净不了（大头在 fe-core 通用节点 + 带副作用；连接器侧 EXPLAIN-only 切片因 `ConnectorSession` 无 explain 信号做不干净）。**用户拍板跳过**。详见 tasklist 该条。 +- **PERF-11 部分（`10b7d29423f`，iceberg）= C12 + C15(a) + C13(plan)**：`buildRange` 穿 1 条目 `PerFileScratch`（键 `task.file()`），partition JSON/identity/delete 载体 per-slice→per-file + 同文件 k range 共享不可变实例 + v3 supply put per-file。3 视角对抗 byte-parity 0 破坏；全模块 1064 pass/1 skip。设计/小结/红队见 `designs/FIX-PERF-11-*`。 --- -# 🆕 下一个 session = **P2 剩余（PERF-09 / PERF-10 / PERF-11 三选一）** +# 🆕 下一个 session = **P2 剩余（PERF-09 框架层 / PERF-11 remainder 二选一）** -> 剩三项，均 P2（CPU/payload/框架层，触发门槛较高但模式典型）。**开场先按 README 流程 step 4 向用户复述并让其定下一个做哪个**（下面给推荐 + 各自要点；**行号信 grep**，PERF-07/08 后可能又漂）。 - -## 推荐先做：PERF-10（C8）——WHERE conjunct 每查询转换 5~6 次 -- **轻活、iceberg-local、与已完成的 PERF-01 同一失效点**（node 字段 memo，宜顺手）。 -- **病灶**：同组 conjunct 在 `buildRemainingFilter`(×3) + `buildScan` + `getScanNodeProperties`（EXPLAIN 专用序列化，非 EXPLAIN 也跑）被 `convertPredicate` 转换 5~6 次/查询。单次微秒~毫秒，与簇1 同源叠加。 -- **修复方向**：node 字段 memo，与 `cachedPropertiesResult` **同点失效**（复用 PERF-01 立住的失效收窄模式）。 -- **⚠ 落地前 grep**：`IcebergScanPlanProvider` 里 `convertPredicate` 的 5~6 个调用点真实位置 + 各自输入是否恒等（同一 conjunct 列表）→ 确认可安全 memo（同输入同输出、谓词不变量）。 +> 剩两块。**开场先按 README 流程 step 4 向用户复述并让其定做哪个**（下面给要点；**行号信 grep**，PERF-08/11 后必漂）。PERF-10 已暂缓、不再是候选。 ## PERF-09（C5）——streaming pump 逐 split 重建 backend 候选集（**fe-core 框架层**） -- **⚠ 改的是 fe-core 通用框架**（惠及所有连接器、非 source-specific，**允许改**）；但属共享热路径，须证 **byte + cost 对所有连接器双不变**（对齐「共享框架须双不变」纪律 + 保持 connector-agnostic）。 +- **⚠ 改的是 fe-core 通用框架**（惠及所有连接器、非 source-specific，**允许改**）；但属共享热路径，须证 **byte + cost 对所有连接器双不变**（对齐「共享框架须双不变」纪律 + 保持 connector-agnostic）。**需两段验**（碰 fe-core）。 - **病灶**：`startStreamingSplit:~1638` 逐 split `addToQueue` → `SplitAssignment` 每 split 重建 backend 候选集（`FederationBackendPolicy.computeScanRangeAssignment` 全量 backend 拷贝 + shuffle + multimap）+ `synchronized` 往返。10⁵~10⁶ split × ~100 BE ≈ 10⁷~10⁸ 冗余。 - **修复方向**：pump 侧微批（64~256/批）。**先 grep** 该泵 + `SplitAssignment` 现结构，确认微批不改分配语义（同一 split 集合的 backend 分配结果不变）。 -## PERF-11（簇5，C12 C13 C14 C15）——per-split 不变量 + payload 放大（**批量、可拆多 commit**） -- 同一 `buildRange`/`populateRangeParams` 区域：C12 每 byte-slice 重算 partition JSON/identity map/delete 转换 →(specId,PartitionData) memo；C13 v3 delete 双重 thrift 转换 → 转一次 per-file 复用；C14 通用节点 per-split `LocationPath.of`/造完即弃的 columns-from-path/`getFileCompressType` → hoist（**⚠ 涉 fe-core 通用节点，保持 connector-agnostic**）；C15 delete 列表+partition JSON 逐 slice 复制进 `TFileRangeDesc` → per-file 共享引用。 -- **先 grep** `buildRange` / `populateRangeParams` / `IcebergScanRange` 真实结构再逐条立项。 +## PERF-11 remainder（C15b 线路字节 / C13-wire / C14）——**已部分完成，剩下三块各有前置** +- **C15(b) 线路字节 + C13(wire)**：`IcebergScanRange.populateRangeParams`（现 `:330`，行号信 grep）逐 range 把完整 delete 列表 `delete.toThrift()` + partition JSON 塞进每个 `TFileRangeDesc`。真正减线路字节须 **thrift params 级 delete-file 字典 + per-range 索引 + BE reader 改** = **协议演进（FE+BE 双向 + 兼容性），非回归修复**（审计 C15 自述）——**立项前须与用户确认是否要做这个大改**。 +- **C14 fe-core 通用节点 per-split hoist**：`FileQueryScanNode`/`PluginDrivenScanNode` 逐 split `LocationPath.of`（URLEncoder+URI.create）、每 split 新建 provider（`getFileCompressType`）、造完即弃的 columns-from-path（被 `IcebergScanRange.populateRangeParams` unset）。**⚠ 框架层**、量级较小（~几百 ms@10 万 split）、须证 byte+cost 双不变。与 PERF-09 同属框架层，可一并评估。 ## 铁律提醒 -- PERF-10/11(iceberg 部分) 修复放**连接器侧** node 字段/memo，不碰 fe-core（对齐 PERF-01 模式）。 -- PERF-09 与 PERF-11 的 C14 碰 **fe-core 通用节点/框架**：允许改（跨连接器普惠）但**禁按源名分支** + 须证 byte+cost 双不变。 -- 立项先读审计报告 §2/§5 对应簇原始证据（findings.json 有完整调用链），再对照 HEAD 真实代码 review（HANDOFF 行号/方案可能过时）。 +- PERF-11 remainder 的 C15b 碰 **BE + thrift 协议**：立项前必与用户确认（大改、非回归修复）。 +- PERF-09 与 PERF-11 的 C14 碰 **fe-core 通用节点/框架**：允许改（跨连接器普惠）但**禁按源名分支** + 须证 byte+cost 双不变 + **两段验**。 +- 立项先读审计报告 §2/§5 对应簇原始证据（findings.json 有完整调用链），再对照 HEAD 真实代码 review（HANDOFF 行号/方案可能过时）。PERF-11 已实证：C13 原证据（`IcebergRewritableDeleteStash`）PERF-07 已删——**动前必 grep 核实证据未 stale**。 --- diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-per-file-invariant-memo-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-per-file-invariant-memo-design.md new file mode 100644 index 00000000000000..82584e71b1cbc6 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-per-file-invariant-memo-design.md @@ -0,0 +1,90 @@ +# FIX-PERF-11 — 大扫描 per-split 不变量重算 + payload 复制（C12 / C15a / C13-plan）设计 + +> 硬约束：性能修复 → 行为逐字节不变（同样的 range、同样的 thrift、同样的扫描结果）。只减 planning CPU + FE heap。 +> 基线：分支 `catalog-spi-review-16`，行号已按 HEAD 重新 grep（PERF-01~08 后）。 + +--- + +## Problem + +一个数据文件被 `TableScanUtil.splitFiles` 切成 **k 个 byte-slice**（k 可达几十），每个 slice 是一个 `FileScanTask`。规划时对**每个 slice** 都调一遍 `buildRange`（`IcebergScanPlanProvider.buildRange:1126`），重算一遍**对该文件恒定**的量： +- `partitionDataJson`（`IcebergPartitionUtils.getPartitionDataJson` = 日期/时间格式化 + Jackson `writeValueAsString`，最贵）； +- identity map + ordered `partitionValues`（`getIdentityPartitionInfoMap` + LinkedHashMap 重排）； +- delete 载体列表（`buildDeleteFiles` → 逐 delete `convertDelete`，含 bounds 解码）。 + +**10 万 slice 的 timestamp 分区 MOR 表 ≈ 0.5~2 秒额外 planning CPU + 大量垃圾**（审计 C12）。此外 k 个 slice 的 `IcebergScanRange` 各持一份这些量的独立实例 → FE heap 放大（C15a）。v3 时 `buildRangeForTask` 还**逐 slice** 调 `rewritableDeleteDescs()` + `rewritableDeleteSupply.put()`（同一文件 k 次幂等 put，C13 plan 侧）。 + +## Root Cause + +`buildRange` 的输出只有 `start` / `length` / `selfSplitWeight`（= `task.length()` + Σdelete 大小）是 **per-slice** 的；其余全部是 **per-file 不变量**（由 `task.file()`/`task.deletes()`/scan 级 zone·spec 决定）。但没有任何 memo，逐 slice 全量重算。`task.file()` 对一个文件的全部 slice 返回**同一 `DataFile` 实例**（`SplitScanTask` 委派），且 `splitFiles` 把一个文件的 slice **连续**吐出——天然可按文件 memo。 + +## Design（连接器侧，单一 choke point） + +`buildRangeForTask`（`:739`）是 eager 循环（`:681`）与流式 `IcebergStreamingSplitSource`（`:564`）**共用**的唯一入口。在它这里穿入一个 **1 条目「当前文件」缓存**（consecutive-file cache）： + +``` +// 伪码：per-file 不变量缓存，穿过 buildRangeForTask +class PerFileScratch { + DataFile file; // 身份键（== 比较） + Integer partitionSpecId; + String partitionDataJson; + Map partitionValues; // unmodifiable，跨 slice 共享同一实例 + List deleteCarriers; // 跨 slice 共享同一实例 + String fileFormat; Long firstRowId; Long lastUpdatedSequenceNumber; + String rawDataPath; String normalizedPath; +} +``` + +`buildRange(slice)`：若 `task.file() == scratch.file` → 复用上面全部 per-file 量，只用 slice 的 `start/length` 重算 `selfSplitWeight` 并组装 range；否则重算 per-file 量、刷新 scratch。**range 直接引用 scratch 里的 `partitionValues`/`deleteCarriers` 同一（不可变）实例** → 同文件 k 个 range 共享实例（C15a：省 heap/GC，`IcebergScanRange` 全 final + `unmodifiableMap/List`，只读，安全共享）。 + +- **eager 路径**：scratch 是 `planScanInternal` 里的一个局部，穿进循环。 +- **流式路径**：scratch 是 `IcebergStreamingSplitSource` 的一个实例字段。**仍 1 条目 → O(1) 内存**，不破 PERF-04 的流式不物化/防 OOM 纪律。 + +**C13 plan 侧**：`buildRangeForTask` 里 `rewritableDeleteSupply.put()` 改为**仅当文件切换时**（即该文件第一个 slice）做一次——同一文件 k 次幂等 put 坍缩为 1 次；put 的 value（`rewritableDeleteDescs()`）也从 per-file 共享 deleteCarriers 派生一次。行为不变（幂等 put 相同 value）。 + +**为什么行为逐字节不变**：per-file 量只依赖 `task.file()`/`task.deletes()`/scan 级 zone·spec，对同文件全部 slice 恒等；1 条目缓存因 slice 连续而 100% 命中同文件；range 的 per-slice 字段（start/length/weight）仍逐 slice 真实计算。→ 产出的 range 列表与 thrift 逐字节一致。 + +## 明确不做（本次范围外，另议） + +- **线路字节去重（C15b）**：`populateRangeParams`（`IcebergScanRange:330`）**逐 range** 把该文件完整 delete 列表 `toThrift()` + partition JSON 塞进每个 `TFileRangeDesc`——这是**线路协议**逐 range 自带一份的固有形状。真正减线路字节要改 thrift（params 级 delete-file 字典 + per-range 索引）+ **BE reader**，是协议演进、非回归修复（审计 C15 自述）。本次不碰。 +- **fe-core 通用节点 per-split hoist（C14）**：`FileQueryScanNode`/`PluginDrivenScanNode` 逐 split 的 `LocationPath.of`、逐 split 新建 provider、造完即弃的 columns-from-path。属**框架层**（惠及所有连接器但须证 byte+cost 双不变），且量级更小（~几百 ms@10 万 split）。单列，另行决策。 +- **跨文件分区元组 memo**：审计「可选」的 `(specId, partition tuple)` 二级 memo（不同文件共享分区值时再 dedup partition JSON）——需 `StructLikeWrapper` map、**无界内存**，破流式 OOM 安全。跳过；如需，仅限 eager 路径且加界。 + +## Implementation Plan（已落地） + +一个连接器侧 `[perf]` commit（仅 `IcebergScanPlanProvider.java` + 测试）： +1. 新增私有静态类 `PerFileScratch`（10 个 per-file 字段，`file` 为身份键）。 +2. 抽 `computePerFileInvariants(...)`（原 buildRange 前半段逐字不变，末尾写入 scratch）；`buildRange` 改为「`scratch.file == dataFile` ? 复用 : compute」+ per-slice `start/length/selfSplitWeight` + 组装。 +3. `buildRangeForTask` 加 `scratch` 参；**动 buildRange 前**捕获 `firstSliceOfFile = scratch.file != dataFile`，`rewritableDeleteSupply.put` 仅在 `firstSliceOfFile` 时（覆盖含最后一个文件）。 +4. eager `planScanInternal` 建局部 `PerFileScratch` 穿入；流式 `IcebergStreamingSplitSource` 加 `private final PerFileScratch scratch`（每 source = 每 scan、单线程 → O(1)）。 +5. `planCountPushdown` 的 `buildRange` 传 `null` scratch（单 range、不缓存）。 +6. 度量守门：`@VisibleForTesting int perFileInvariantComputeCount`（`computePerFileInvariants` 首行 `++`）。 + +## Risk Analysis + +- **正确性关键**：per-file 量若误缓存会串味（错分区 JSON/错 delete = 错扫描结果/静默漏删）。缓解=键用 `task.file()` 身份 `==`、只在同文件复用、per-slice 字段照算；红队专攻 parity。 +- **流式 OOM**：1 条目缓存 O(1)，不累积。 +- **实例共享**：`IcebergScanRange` 全 final + unmodifiable，`populateRangeParams` 只读 → 共享安全（不会有 range 改动共享 list）。 +- **v3 put-once**：幂等 put 相同 value，坍缩为 1 次不改 supply 内容。 + +## Test Plan + +### Unit Tests +- **per-file 复用度量守门**：一个数据文件切多 slice（构造 ≥2 slice），断言 `getPartitionDataJson`/`getIdentityPartitionInfoMap`（或一个连接器内计数器）对该文件**恰算 1 次**，而非 k 次。 +- **byte-parity**：memo 前后同一表 planScan 产出的 range 列表逐字段等价（path/original/start/length/partitionSpecId/partitionDataJson/partitionValues/deleteFiles/weight），尤其**多 slice 文件的每个 slice 的 start/length 各异、其余共享**。 +- **实例共享**：同文件多 slice 的 range 的 `getPartitionValues()`/delete 载体是**同一实例**（`assertSame`）。 +- **v3 put-once**：rewritableDeleteSupply 对一个文件只被 put 一次、且 value 与逐 slice 版一致。 +- 流式路径同构：`IcebergStreamingSplitSource` 产出与 eager 逐字节一致（既有 parity 测试扩展）。 + +### E2E Tests +- 大 MOR 表 + 分区 + 多 slice 的 SELECT/COUNT 结果 parity 随 P6.6 切换阶段统一补（对齐既有约定）。 + +## 设计红队（3 独立对抗视角，byte-parity 专攻） + +三视角（不变量真实性 / 键与顺序 / 实例共享+put-once）均判 `parity_preserved=true`，无法证伪。关键确认：不同逻辑文件**永不共享** `DataFile` 实例（SDK `ContentFileUtil.copy` + manifest 缓存 `dataFile.copy()` 逐条拷贝，后者带「防复用」注释），故身份键不会跨文件返回 stale；`splitFiles`（`FluentIterable.transformAndConcat`）连续吐同文件 slice，1 条目缓存 100% 命中，即便假设非连续也只降 perf 不破 correctness；`IcebergScanRange` 全 immutable、消费者只读、刷新是换引用不改旧对象 → 流式 look-ahead + 已发出 range 保持各自正确对象。**3 条落地纪律（已遵守）**：① `selfSplitWeight/start/length` 必须 per-slice（只 delete-size 和是 per-file，但 length 项 per-slice，故整权重 per-slice）；② put 必须在**文件第一个 slice（miss）用新文件路径**（否则漏最后一个文件 → v3 静默复活）；③ scratch 每 scan 新建、绝不 provider/static 字段。 + +## 度量收益 + +- **C12**：partition JSON / identity / delete 载体从 **per-slice(k)** → **per-file(1)**；10 万 slice ≈ 省 0.5~2s planning CPU + 大幅减垃圾。 +- **C15a**：同文件 k 个 range 共享 `partitionValues`/delete 载体实例 → 省 FE heap。 +- **C13(plan)**：v3 rewritable-delete supply put 从 per-slice → per-file。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-per-file-invariant-memo-summary.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-per-file-invariant-memo-summary.md new file mode 100644 index 00000000000000..014eec10fa27a1 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-per-file-invariant-memo-summary.md @@ -0,0 +1,35 @@ +# FIX-PERF-11 — 大扫描 per-file 不变量 memo（C12 / C15a / C13-plan）小结 + +> 设计 + 红队详见 [`FIX-PERF-11-per-file-invariant-memo-design.md`](./FIX-PERF-11-per-file-invariant-memo-design.md)。 +> 一个连接器侧 commit。硬约束达成：只减 planning CPU + FE heap，扫描 range/thrift/结果逐字节不变。 + +## Problem + +一个数据文件被 `splitFiles` 切成 k 个 byte-slice，`buildRange` 逐 slice 重算**对整个文件恒定**的量（partition JSON 的 Jackson+时区格式化、identity/ordered 分区值、delete 载体转换）。10 万 slice 的分区 MOR 表 ≈ 多花 0.5~2 秒 planning CPU + 大量垃圾；k 个 range 各持独立副本放大 FE heap；v3 时还逐 slice 幂等 put 删除供给。 + +## Root Cause + +`buildRange` 输出仅 `start/length/selfSplitWeight` 是 per-slice，其余全是 per-file 不变量（由 `task.file()`/`task.deletes()`/scan 级 zone·spec 决定），但无 memo。`task.file()` 对一个文件全部 slice 返回同一 `DataFile` 实例，且 slice 连续吐出。 + +## Fix（`10b7d29423f`） + +在共用 choke point `buildRangeForTask` 穿入 **1 条目「当前文件」缓存** `PerFileScratch`： +- 抽 `computePerFileInvariants`（原逐 slice 计算逐字不变）；`buildRange` 改「`scratch.file == dataFile` ? 复用 : compute」+ per-slice `start/length/selfSplitWeight` + 组装。**range 直接引用 scratch 的不可变 `partitionValues`/`deleteCarriers` 实例** → 同文件 k range 共享（C15a）。 +- eager `planScanInternal` 局部 scratch；流式 `IcebergStreamingSplitSource` 实例字段 scratch（每 scan 新建、单线程 → **O(1) 内存、不破流式防 OOM**）。 +- **C13 plan**：`rewritableDeleteSupply.put` 改为**文件第一个 slice（miss）时用新文件路径**一次（覆盖含最后一个文件，杜绝 v3 静默复活）。 +- `planCountPushdown` 传 `null` scratch（单 range 不缓存）。 +- 度量守门 `@VisibleForTesting int perFileInvariantComputeCount`。 + +**逐字节不变**靠：per-file 量只依赖 `task.file()`/scan 不变量、对同文件恒等；不同逻辑文件永不共享 `DataFile` 实例（copy-per-entry）→ 身份键不 stale；slice 连续 → 1 条目 100% 命中；per-slice 字段照算。3 独立对抗视角均判 parity 保持、0 破坏性发现。 + +## Tests + +`IcebergScanPlanProviderTest` +2：① 96MB 分区文件按 32MB 切 3 slice → `perFileInvariantComputeCount==1`、各 slice partition 值/JSON 相同而 start/length 连续平铺；② 两个切分文件 → count==2 且各带自己的分区值（无跨文件串味）。既有 103 scan-plan parity 测试 + 66 transaction（delete 供给）全绿。 + +## Result + +- **C12**：partition JSON/identity/delete 载体 per-slice(k) → per-file(1)，10 万 slice 省 0.5~2s CPU + 减垃圾。 +- **C15a**：同文件 k range 共享 `partitionValues`/delete 载体实例，省 FE heap。 +- **C13(plan)**：v3 rewritable-delete supply put per-slice → per-file。 +- **线路字节（C15b）+ fe-core 通用节点 hoist（C14）本次不做**（协议演进/框架层，另议）。 +- 验证：iceberg `IcebergScanPlanProviderTest` 105/105 + `IcebergConnectorTransactionTest` 66/66 + 全模块 `1064 pass / 1 skip / 0 fail`，BUILD SUCCESS + 0 checkstyle。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index 3c4083969cd2ef..86fd3ec7fc59c5 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -130,3 +130,16 @@ - **踩坑**：`ActionTestTables` 起初误加未命名的 `DeleteFile` import（builder 内联进 `addDeletes` 不具名）→ checkstyle UnusedImports；删 import + 删一个凑数的 anchor 方法（违简单律）。 - **结果**：fe-core BUILD SUCCESS + `ConnectorRewriteDriverTest` 5/5 + 0 checkstyle；iceberg BUILD SUCCESS + `IcebergExpireSnapshotsActionTest` 9/9 + 0 checkstyle。度量：C19 整表 `planFiles()` G+1→1；C21 delete manifest 远程读 O(S×M)→O(distinct M)。 - **下一步**：见 HANDOFF —— P2 剩余（PERF-09 fe-core streaming pump 微批 / PERF-10 WHERE conjunct memo / PERF-11 簇5 per-split 不变量+payload）。 + +--- + +## 2026-07-18 — session 3（续）：PERF-10 复核判低价值暂缓 + PERF-11 部分完成（commit `10b7d29423f`） + +- **PERF-10（C8，WHERE conjunct 转换）复核后暂缓（🔬）**：按 findings.json #7 权威口径 + HEAD 核对——两个独立 verifier 均判 **Low**（纯 planning CPU 微秒~低毫秒、无远程 IO），且当初「每次转换旁跟一次昂贵 loadTable」已被 PERF-01/07 修掉，单独价值极低；干净不了（k≈7 中 4 次在 fe-core 通用节点 `buildRemainingFilter`、**带副作用** `filteredToOriginalIndex`，须双不变 + 缓存副作用）；连接器侧「安全小切片」（去 `getScanNodeProperties` 的 EXPLAIN-only `PUSHDOWN_PREDICATES_PROP` 白干）也做不干净——`ConnectorSession` 无 explain 信号（加=碰框架 SPI，用户不碰）、复用 `buildScan` 转换又因 `getScanNodeProperties` 早于 `buildScan` 顺序反了。**用户 2026-07-18 拍板跳过、改做高价值 PERF-11**。tasklist 标 🔬 记录复活条件（愿加中性 explain 信号则连接器侧可复活）。 +- **PERF-11 部分完成（`10b7d29423f`，连接器侧一个 commit）= C12 + C15(a) + C13(plan)**： + - **调研**：读 `buildRange`/`buildRangeForTask`(共用 choke point，eager+流式)/`IcebergScanRange.populateRangeParams`(全 immutable、只读)/`IcebergStreamingSplitSource`(单线程、每 source=每 scan)；确认 per-file 不变量 = partition JSON/identity/ordered 值/delete 载体/format/row-lineage/normalized path，per-slice = start/length/`selfSplitWeight`(=length 项 per-slice)。**C13 原证据 stale**（`IcebergRewritableDeleteStash` PERF-07 已删；现供给走 `IcebergStatementScope.rewritableDeleteSupply` 每语句 map，per-slice 幂等 put）。 + - **设计红队（3 视角 byte-parity 专攻）**：均 `parity_preserved=true`、0 破坏。关键确认：不同逻辑文件永不共享 `DataFile` 实例（SDK `ContentFileUtil.copy`+缓存 `dataFile.copy()` 逐条拷贝，后者带防复用注释）→ 身份键不 stale；`splitFiles`(`transformAndConcat`) 连续吐同文件 slice → 1 条目缓存 100% 命中、即便非连续也只降 perf；immutable+只读+换引用 → 共享/流式 look-ahead 安全。**3 落地纪律**：selfSplitWeight/start/length 必 per-slice；put 在文件首 slice(miss) 用新文件路径(覆盖末文件、防 v3 复活)；scratch 每 scan 新建非 provider/static。 + - **落地**：`PerFileScratch`(10 字段)+`computePerFileInvariants`(原逐 slice 逐字不变、末尾写 scratch)；`buildRange` 复用/compute + per-slice 组装、range 引用 scratch 不可变 `partitionValues`/`deleteCarriers` 实例；`buildRangeForTask` 动 buildRange 前捕获 `firstSliceOfFile`、put 仅首 slice；eager 局部 scratch、流式实例字段 scratch；count-pushdown 传 null；`@VisibleForTesting perFileInvariantComputeCount` 守门 + 2 测试(切分文件算 1 次/两文件算 2 次无跨文件串味)。 + - **明确不做**（tasklist 记）：**C15(b) 线路字节**（`populateRangeParams` 逐 range `toThrift()`+partition JSON 塞每个 `TFileRangeDesc`，减字节须 thrift 字典+BE reader 改=协议演进非回归修复）、**C13(wire) 二次转换**（同处线路侧、随 C15b）、**C14 fe-core 通用节点 hoist**（框架层、须双不变）。 + - **结果**：`IcebergScanPlanProviderTest` 105/105（+2）、`IcebergConnectorTransactionTest` 66/66、**全模块 1064 pass/1 skip/0 fail**、BUILD SUCCESS + 0 checkstyle。 +- **下一步**：见 HANDOFF —— PERF-09（fe-core streaming pump 微批）或 PERF-11 remainder（C15b/C13-wire/C14）。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index 28e492a8b46c8d..f8107aa0e20ef3 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -24,8 +24,8 @@ | PERF-07 | P1 | C20 | 一条 DML 重复 load 同表 → 每语句「表加载归属者」(`ConnectorStatementScope` 挂 StatementContext) 读写共享一次加载；拆胖句柄；delete stash 下沉到每语句作用域(删单例+类)；beginWrite 取共享表 | — | ✅ 完成 | `97bdcd6bdbe`+`ea7fd1f6e7a` | | PERF-08 | P2 | C19 C21 | 维护路径逐单位重扫 / 无去重（rewrite_data_files 每 group planFiles；expire_snapshots S×M）→ union 一次注册 / 按 path 去重 | 改动小可先行 | ✅ 完成 | `89cc39c8d88`(C19)+`be0035eff62`(C21) | | PERF-09 | P2 | C5 | **（fe-core 框架层）** streaming pump 逐 split 重建 backend 候选集 + 锁往返 → 微批 | ⚠ 跨连接器，见约束 | ⏳ | | -| PERF-10 | P2 | C8 | 同组 WHERE conjunct 被转换 5~6 次/查询 → node 字段 memo（与 01 同点失效） | 与 01 同源 | ⏳ | | -| PERF-11 | P2 | C12 C13 C14 C15 | per-split 不变量重算 + payload 逐 slice 复制 → hoist / (specId,PartitionData) memo / per-file 共享 | 同区域批量 · ⚠ 含 fe-core 通用节点 | ⏳ | | +| PERF-10 | P2 | C8 | 同组 WHERE conjunct 被转换 5~6 次/查询 → node 字段 memo（与 01 同点失效） | 与 01 同源 | 🔬 暂缓（复核判低价值） | | +| PERF-11 | P2 | C12 C13 C14 C15 | per-split 不变量重算 + payload 逐 slice 复制 → hoist / (specId,PartitionData) memo / per-file 共享 | 同区域批量 · ⚠ 含 fe-core 通用节点 | 🚧 部分完成（C12+C15a+C13-plan） | `10b7d29423f` | --- @@ -94,13 +94,23 @@ - **修复方向**：pump 侧微批（64~256 个/批）。 - **约束**：⚠ 改的是 **fe-core 通用框架**（惠及所有连接器，非 source-specific，允许改）；但属共享热路径，须证 **byte + cost 对所有连接器双不变**（对齐现有"共享 MVCC 方法须双不变"纪律）。 -### [ ] PERF-10 — C8：WHERE conjunct 转换 5~6 次/查询 +### [🔬] PERF-10 — C8：WHERE conjunct 转换 5~6 次/查询 · 暂缓（2026-07-18 复核判低价值，用户拍板跳过） - **病灶**：同组 conjunct 在 `buildRemainingFilter`（×3 处）+ `buildScan:974` + `getScanNodeProperties:1428`（EXPLAIN 专用序列化，**非 EXPLAIN 也跑**）被转换 5~6 次。单次微秒~毫秒级，但与簇1 同源叠加。 -- **修复方向**：node 字段 memo，与 `cachedPropertiesResult` 同点失效。 +- **修复方向（原）**：node 字段 memo，与 `cachedPropertiesResult` 同点失效。 +- **🔬 复核结论（暂缓，findings.json #7 权威口径 + HEAD 核对）**： + - **价值现已很低**：两个独立 verifier 均判 **Low**——纯 planning 线程 CPU 微秒~低毫秒、**无远程 IO**；当初记它是因「每次转换旁各跟一次昂贵远程 loadTable」，而那些 loadTable 已被 **PERF-01/07 全部修掉**，单独看只剩微秒级 CPU。 + - **且干净不了**：k≈7 中 **4 次在 fe-core 通用节点**（`PluginDrivenScanNode.buildRemainingFilter:1888`，且**带副作用** `filteredToOriginalIndex`，`pruneConjunctsFromNodeProperties`/`getSplits` 会读）——memo 须连副作用一起缓存 + 证「对所有连接器字节+成本双不变」，为微秒收益动共用热路径不划算。 + - **连接器侧「安全小切片」也做不干净**：`getScanNodeProperties:1518-1531` 的 `PUSHDOWN_PREDICATES_PROP` 转换+拼串是 **EXPLAIN 专用**（唯一消费者 `appendExplainInfo:1633`），普通查询每次白干；但要 explain-gate，连接器需知「是否 EXPLAIN」而 `ConnectorSession` **无此信号**（加 = 碰框架 SPI，用户明确不碰）；复用 `buildScan` 转换又因 `getScanNodeProperties`（节点初始化期）**早于** `buildScan`（规划期）而顺序反了、不可行。 + - **处置**：用户 2026-07-18 拍板**跳过**，改做高价值 PERF-11。若将来愿加一个中性 `ConnectorSession` explain 信号，连接器侧 explain-gate 可复活（收益仍微秒级）。 - **依赖**：与 PERF-01 同源，宜在 01 之后顺手。 -### [ ] PERF-11 — 簇5：per-split 不变量 + payload 放大（C12 C13 C14 C15） -> 同一 `buildRange`/`populateRangeParams` 区域的 per-split 重复计算与 payload 放大，作为一个任务批量处理（可拆多个 commit）。 +### [🚧] PERF-11 — 簇5：per-split 不变量 + payload 放大（C12 C13 C14 C15） · 部分完成 `10b7d29423f` +> 同一 `buildRange`/`populateRangeParams` 区域的 per-split 重复计算与 payload 放大，拆多 commit。权威设计/小结/红队见 [`designs/FIX-PERF-11-per-file-invariant-memo-*`](./designs/FIX-PERF-11-per-file-invariant-memo-design.md)。 +> **已完成（`10b7d29423f`，连接器侧）= C12 + C15(a) + C13(plan 侧)**：`buildRange` 穿 1 条目 `PerFileScratch`（键 `task.file()` 身份），partition JSON/identity/ordered 值/delete 载体 per-slice→per-file；同文件 k range 共享不可变实例（省 FE heap）；v3 `rewritableDeleteSupply.put` per-slice→per-file（文件首 slice、覆盖末文件）。eager+流式共用 `buildRangeForTask` choke point，流式 scratch O(1) 不破 OOM。3 视角对抗 byte-parity 0 破坏；全模块 1064 pass/1 skip。 +> **仍未做（deferred，另立 commit/任务）**： +> - **C15(b) 线路字节**：`populateRangeParams` 逐 range 把完整 delete 列表 `toThrift()` + partition JSON 塞进每个 `TFileRangeDesc` —— 真正减线路字节须 thrift params 级 delete-file 字典 + per-range 索引 + **BE reader 改**，是**协议演进非回归修复**（审计 C15 自述）。 +> - **C13(wire) 二次转换**：上面的 `populateRangeParams` per-range `delete.toThrift()` 亦是 C13 的「第二次转换」——与 C15b 同一处线路侧，随 C15b 一并处理或单独 hoist（缓存 per-file 已转换 descs，牵扯持有 thrift 对象的内存权衡）。 +> - **C14 fe-core 通用节点 hoist**：见下（框架层，须双不变，另议）。 - **C12**：一个 data file 被 `TableScanUtil.splitFiles` 切成 k 个 byte-slice 后，`buildRange:1045` 对每个 slice 重算 partition JSON（Jackson 序列化 + 时区格式化）、identity map、delete 转换 —— (specId, PartitionData) 级不变量。100k split ≈ 0.5~2s CPU。修：按 (specId, PartitionData) memo。 - **C13**：v3 rewritable-delete stash：同一 delete 列表 plan 期 `rewritableDeleteDescs:302-313` 转一次 thrift，`populateRangeParams` 再转一次；每 slice 重复 put 相同 supply。修：转一次复用、per-file 而非 per-slice。 - **C14**：通用节点 per-split 重复 `LocationPath.of`（URLEncoder+URI.create）、重建 columns-from-path 却被 `IcebergScanRange.populateRangeParams:435-437` unset 丢弃；`resolveScanProvider` 每 split 经 `getFileCompressType` 反复解析。修：hoist 不变量 / 删造完即弃分支。⚠ 涉 **fe-core 通用节点**——保持 connector-agnostic，勿加 source-specific 分支。 From 74162180a794dcdab037db17d6041ff696585581 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 07:31:10 +0800 Subject: [PATCH 25/47] [perf](catalog) fe-core: memoize PluginDrivenScanNode scan-provider per handle (PERF-11 / C14) Root cause: PluginDrivenScanNode.resolveScanProvider() = connector.getScanPlanProvider(currentHandle), and the SPI contract builds providers fresh/stateless per call. It is invoked on the per-split / per-range hot path (getFileCompressType per split, getDeleteFiles per range, plus ~10 planning sites), so every split re-allocated a scan provider + did two TCCL classloader swaps to run what is an identity no-op for iceberg. All three enumeration paths hit it (eager coordinator loop, concurrent partition-batch appendBatch threads, single streaming pump). Lowest-magnitude tier (~100-200ns/split, no remote IO), but pure waste: the provider is a function of currentHandle, which is only refined during early pushdown/pin, before split enumeration. Fix: memoize the resolved provider keyed on currentHandle IDENTITY, held in an immutable (handle, provider) holder published via a single volatile write. A hit reuses the instance; an identity miss (first call, or currentHandle swapped by pushdown/pin) re-resolves and republishes. A null provider (no scan capability) is cached and returned correctly. Because it is keyed on the exact currentHandle each caller would pass, it is byte-identical to calling connector.getScanPlanProvider(currentHandle) every time - no reliance on any connector's provider-selection semantics. The final fields + single volatile write make it safe for the concurrent partition-batch appendBatch threads (no torn new-key/old-provider read); sharing one provider instance across the scan is already the established pattern on the heavier planScan path (startSplit/startStreamingSplit capture and share one provider across concurrent async tasks). Connector-agnostic (no source branching); connectors that field-cache their provider (jdbc/maxcompute) see a strict no-op, connectors that new-per-call (iceberg/paimon/hive) lose the per-split allocation - no connector regresses. Measurement / gate: PluginDrivenScanNodeScanProviderSelectionTest. memoizesProviderForStableHandleAndReResolvesOnHandleChange asserts getScanPlanProvider is invoked exactly once for a stable handle across repeated resolves (was N-per-split; observed red "Wanted 1 time ... but was 3" before the fix) and re-resolves once on a handle change. Full PluginDrivenScanNode*Test family: 103 pass / 0 fail / 0 skip; fe-core Checkstyle 0 violations. fe-core-only change; the iceberg connector does not depend on fe-core and is unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../datasource/scan/PluginDrivenScanNode.java | 34 ++++++++++++++- ...ivenScanNodeScanProviderSelectionTest.java | 43 +++++++++++++++++++ 2 files changed, 76 insertions(+), 1 deletion(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index 050c75c796872f..80d9c3e381a4d2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -177,6 +177,10 @@ public class PluginDrivenScanNode extends FileQueryScanNode { // Maps filtered conjunct indices (after CAST removal) back to original conjunct indices private List filteredToOriginalIndex; + // Memoized resolveScanProvider() result, keyed on currentHandle identity. See resolveScanProvider(). + // Volatile so the concurrent partition-batch appendBatch threads read a self-consistent (handle, provider). + private volatile ResolvedScanProvider resolvedScanProvider; + public PluginDrivenScanNode(PlanNodeId id, TupleDescriptor desc, boolean needCheckColumnPriv, SessionVariable sv, ScanContext scanContext, Connector connector, @@ -221,7 +225,35 @@ public static PluginDrivenScanNode create(PlanNodeId id, TupleDescriptor desc, * {@code null} for connectors without scan capability. */ private ConnectorScanPlanProvider resolveScanProvider() { - return connector.getScanPlanProvider(currentHandle); + // Memoized per currentHandle IDENTITY. The provider is a pure function of currentHandle (SPI contract: + // providers are built fresh/stateless per call and the selected provider does not change across a scan), + // so the per-split getFileCompressType / getDeleteFiles hot path reuses one instance instead of + // re-allocating + TCCL-swapping a provider per split. currentHandle is only refined (a new object) during + // early pushdown/pin, before split enumeration; an identity miss re-resolves — making this byte-identical + // to calling connector.getScanPlanProvider(currentHandle) every time. The immutable holder is published via + // one volatile write so the concurrent partition-batch appendBatch threads always read a self-consistent + // (handle, provider) pair; a null provider (no scan capability) is cached and returned correctly. + ResolvedScanProvider cached = resolvedScanProvider; + if (cached == null || cached.handle != currentHandle) { + cached = new ResolvedScanProvider(currentHandle, connector.getScanPlanProvider(currentHandle)); + resolvedScanProvider = cached; + } + return cached.provider; + } + + /** + * Immutable (handle, provider) pair for {@link #resolveScanProvider()}'s memo. Both fields final so a single + * volatile write of the holder safely publishes the pair to concurrent readers (no torn new-key/old-provider + * view). {@code provider} may be {@code null} for a connector without scan capability. + */ + private static final class ResolvedScanProvider { + private final ConnectorTableHandle handle; + private final ConnectorScanPlanProvider provider; + + ResolvedScanProvider(ConnectorTableHandle handle, ConnectorScanPlanProvider provider) { + this.handle = handle; + this.provider = provider; + } } /** diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeScanProviderSelectionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeScanProviderSelectionTest.java index 23f0ce9e8812f3..ef2c02c847229b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeScanProviderSelectionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeScanProviderSelectionTest.java @@ -66,4 +66,47 @@ public void resolvesProviderForCurrentHandle() { Assertions.assertSame(hiveProvider, Deencapsulation.invoke(node, "resolveScanProvider"), "after the handle changes the node must resolve the matching provider (per-table routing)"); } + + /** + * Guards that {@link PluginDrivenScanNode#resolveScanProvider()} MEMOIZES the resolved provider for a stable + * {@code currentHandle} so the per-split hot path ({@code getFileCompressType} / {@code getDeleteFiles}) stops + * re-allocating a provider on every split, while still RE-RESOLVING when pushdown/pin refines the handle. + * + *

    WHY this matters (Rule 9): providers are built fresh per call (SPI contract), so before this memo + * every split re-ran {@code connector.getScanPlanProvider(currentHandle)} plus a TCCL swap. A mutant that drops + * the memo (resolves per call) reintroduces that O(splits) allocation; a mutant that never re-resolves sends a + * table to a stale provider after the handle is refined. The call-count assertion pins both: exactly ONE resolve + * per distinct handle. Same {@code CALLS_REAL_METHODS} + {@code Deencapsulation} technique as + * {@link #resolvesProviderForCurrentHandle}.

    + */ + @Test + public void memoizesProviderForStableHandleAndReResolvesOnHandleChange() { + PluginDrivenScanNode node = Mockito.mock(PluginDrivenScanNode.class, Mockito.CALLS_REAL_METHODS); + + ConnectorTableHandle icebergHandle = Mockito.mock(ConnectorTableHandle.class); + ConnectorTableHandle hiveHandle = Mockito.mock(ConnectorTableHandle.class); + ConnectorScanPlanProvider icebergProvider = Mockito.mock(ConnectorScanPlanProvider.class); + ConnectorScanPlanProvider hiveProvider = Mockito.mock(ConnectorScanPlanProvider.class); + + Connector connector = Mockito.mock(Connector.class); + Mockito.when(connector.getScanPlanProvider(icebergHandle)).thenReturn(icebergProvider); + Mockito.when(connector.getScanPlanProvider(hiveHandle)).thenReturn(hiveProvider); + Deencapsulation.setField(node, "connector", connector); + + // Stable handle: many resolves (mirroring the per-split getFileCompressType calls) resolve ONCE. + Deencapsulation.setField(node, "currentHandle", icebergHandle); + for (int i = 0; i < 3; i++) { + Assertions.assertSame(icebergProvider, Deencapsulation.invoke(node, "resolveScanProvider"), + "a stable handle must return the memoized provider"); + } + Mockito.verify(connector, Mockito.times(1)).getScanPlanProvider(icebergHandle); + + // Handle refined by pushdown/pin (a NEW handle object): re-resolve exactly once for the new handle. + Deencapsulation.setField(node, "currentHandle", hiveHandle); + for (int i = 0; i < 2; i++) { + Assertions.assertSame(hiveProvider, Deencapsulation.invoke(node, "resolveScanProvider"), + "after the handle changes the node must re-resolve to the matching provider"); + } + Mockito.verify(connector, Mockito.times(1)).getScanPlanProvider(hiveHandle); + } } From 3465f4494b22683e9802f1988256414acfc288d5 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 07:34:30 +0800 Subject: [PATCH 26/47] [doc](catalog) fe-core PERF-11 C14-provider-memo: design + summary + P2-remainder recon; tracker/handoff updates - Design + summary for the resolveScanProvider() per-handle memo (fix committed in 87ff73b1a95). - Records the session-4 recon of the three remaining P2 candidates (all evidence re-verified live, not stale): the streaming-pump micro-batch (fe-core framework, needs sign-off because micro-batching changes the backend split distribution - not byte-identical), the C15b/C13 wire-byte dedup (protocol evolution: thrift delete-file dictionary + BE reader + mixed-version compat, needs sign-off), and the C14 generic-node slices #1/#2/#4 (deferred: cross-connector Hadoop-Path byte risk / new SPI capability surface). User chose only C14 slice #3 (provider memo). - tasklist/HANDOFF: mark C14 provider-memo done; HANDOFF next-step = the two sign-off-gated blocks only. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/perf-hotpath-iceberg/HANDOFF.md | 56 +++++++----- .../FIX-PERF-11-scan-provider-memo-design.md | 87 +++++++++++++++++++ .../FIX-PERF-11-scan-provider-memo-summary.md | 44 ++++++++++ plan-doc/perf-hotpath-iceberg/progress.md | 16 ++++ plan-doc/perf-hotpath-iceberg/tasklist.md | 6 +- 5 files changed, 183 insertions(+), 26 deletions(-) create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-scan-provider-memo-design.md create mode 100644 plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-scan-provider-memo-summary.md diff --git a/plan-doc/perf-hotpath-iceberg/HANDOFF.md b/plan-doc/perf-hotpath-iceberg/HANDOFF.md index ef9c83fd705826..785b423b09bb8d 100644 --- a/plan-doc/perf-hotpath-iceberg/HANDOFF.md +++ b/plan-doc/perf-hotpath-iceberg/HANDOFF.md @@ -6,50 +6,60 @@ --- -# ✅ 已完成（PERF-01 ~ 08 全，PERF-11 部分） +# ✅ 已完成（PERF-01 ~ 08 全，PERF-11 部分含 C14-provider，PERF-10 暂缓） -- **PERF-01~07**：见 `progress.md` + `tasklist.md`（胖 handle + 跨查询 `IcebergTableCache`；分区视图缓存；`file_format_type` 兜底 memoize；manifest cache 两旁路接回；information_schema comment 缓存；vended-cred 每文件 normalizer；每语句 `ConnectorStatementScope` 读写共享一次加载）。 -- **PERF-08（`89cc39c8d88` C19 fe-core + `be0035eff62` C21 iceberg）**：维护命令「逐单位重扫/无去重」——`rewrite_data_files` union 一次登记（全表扫 G+1→1）；`expire_snapshots` 按 `manifest.path()` 去重（O(S×M)→O(distinct M)）。 -- **PERF-10（C8，WHERE conjunct 转换）= 🔬 暂缓**：复核判**低价值**（纯 CPU 微秒、贵的 loadTable 已被 01/07 修掉）且干净不了（大头在 fe-core 通用节点 + 带副作用；连接器侧 EXPLAIN-only 切片因 `ConnectorSession` 无 explain 信号做不干净）。**用户拍板跳过**。详见 tasklist 该条。 -- **PERF-11 部分（`10b7d29423f`，iceberg）= C12 + C15(a) + C13(plan)**：`buildRange` 穿 1 条目 `PerFileScratch`（键 `task.file()`），partition JSON/identity/delete 载体 per-slice→per-file + 同文件 k range 共享不可变实例 + v3 supply put per-file。3 视角对抗 byte-parity 0 破坏；全模块 1064 pass/1 skip。设计/小结/红队见 `designs/FIX-PERF-11-*`。 +- **PERF-01~07**：胖 handle + 跨查询 `IcebergTableCache`；分区视图缓存；`file_format_type` 兜底 memoize；manifest cache 两旁路接回；information_schema comment 缓存；vended-cred 每文件 normalizer；每语句 `ConnectorStatementScope` 读写共享一次加载。 +- **PERF-08（`89cc39c8d88` C19 + `be0035eff62` C21）**：`rewrite_data_files` union 一次登记；`expire_snapshots` 按 `manifest.path()` 去重。 +- **PERF-10（C8）= 🔬 暂缓**：复核判低价值（纯 CPU 微秒、贵的 loadTable 已被 01/07 修掉），用户拍板跳过。 +- **PERF-11 部分**： + - `10b7d29423f`（iceberg）= **C12 + C15(a) + C13(plan)**：`buildRange` 穿 `PerFileScratch`（键 `task.file()`），partition JSON/identity/delete 载体 per-slice→per-file + 同文件 k range 共享不可变实例。 + - `87ff73b1a95`（**fe-core**）= **C14 子项 #3 provider memo**：`PluginDrivenScanNode.resolveScanProvider()` 按 `currentHandle` **身份键** memo（不可变 holder + volatile 发布）——per-split provider 重分配 O(splits)→ 每 handle 1 次。byte-identical、3 视角红队 0 缺陷。设计/小结见 `designs/FIX-PERF-11-scan-provider-memo-*`。 --- -# 🆕 下一个 session = **P2 剩余（PERF-09 框架层 / PERF-11 remainder 二选一）** +# 🆕 下一个 session = **P2 剩余（仅 PERF-09 与 PERF-11 的 C15b/C13-wire 两块，均需用户签字）** -> 剩两块。**开场先按 README 流程 step 4 向用户复述并让其定做哪个**（下面给要点；**行号信 grep**，PERF-08/11 后必漂）。PERF-10 已暂缓、不再是候选。 +> 本 session 已把 3 个候选**全部复核清楚**（recon+对抗 verify，证据均未失效）。剩下两块**都不是干净回归修复、都要用户先拍板**——**开场先按 README step 4 向用户复述这两块 + 各自的签字理由，让其定做哪个或收尾**。C14 只剩低价值/高风险的 #1/#2/#4（暂缓）。行号信 grep。 -## PERF-09（C5）——streaming pump 逐 split 重建 backend 候选集（**fe-core 框架层**） -- **⚠ 改的是 fe-core 通用框架**（惠及所有连接器、非 source-specific，**允许改**）；但属共享热路径，须证 **byte + cost 对所有连接器双不变**（对齐「共享框架须双不变」纪律 + 保持 connector-agnostic）。**需两段验**（碰 fe-core）。 -- **病灶**：`startStreamingSplit:~1638` 逐 split `addToQueue` → `SplitAssignment` 每 split 重建 backend 候选集（`FederationBackendPolicy.computeScanRangeAssignment` 全量 backend 拷贝 + shuffle + multimap）+ `synchronized` 往返。10⁵~10⁶ split × ~100 BE ≈ 10⁷~10⁸ 冗余。 -- **修复方向**：pump 侧微批（64~256/批）。**先 grep** 该泵 + `SplitAssignment` 现结构，确认微批不改分配语义（同一 split 集合的 backend 分配结果不变）。 +## PERF-09（C5）——streaming pump 逐 split 重建 backend 候选集（**fe-core 框架层，需签"接受分配语义变"**） +- **病灶（HEAD 已核）**：`PluginDrivenScanNode.startStreamingSplit:1638-1642` 逐 split `addToQueue` → `SplitAssignment.addToQueue:153`（`synchronized(assignLock)` + `computeScanRangeAssignment`）→ `FederationBackendPolicy.computeScanRangeAssignment:225-311` 每 split 全量 backend 拷贝 + fixed-seed shuffle + multimap + `ResettableRandomizedIterator`，**且 audit 漏算** `equateDistribution:320` O(B log B) 排序 + 两 IndexedPriorityQueue（`enableSplitsRedistribution` 默认 true、无生产 disable 路径→恒跑）。触 iceberg+trino 两流式。 +- **成本**：纯 CPU/GC、无远程 IO，10⁵ split≈0.1-0.5s、10⁶ 几秒。有限、比远程 IO 类小一档。 +- **⚠ 关键**：微批**非字节等价**——攒批令 shuffle+`equateDistribution` 从"单 split 空操作"变"真做均衡"（`nextBe`/`assignedWeightPerBackend` 是持久字段），**同一批 split 落到哪些后端会变**（功能安全：每 split 仍恰分一次、全数据读到，仅"哪个 split 去哪台机器"变，是负载均衡启发式）。且"一次一个"是照搬**上游** legacy `doStartSplit`——改它=在上游基线上演进，非修回归。→ **立项前须用户签"接受这个负载均衡语义变化"**（不能声称行为不变）。 +- **修法**：pump 侧微批（64~256/批），re-check `needMoreSplit()` per 批 + 末批 flush；碰 fe-core → **两段验** + 分配正确性单测（每 split 恰一次/不丢不重/背压）；分配分布变化 + 时延收益须真 BE 分布式验。 -## PERF-11 remainder（C15b 线路字节 / C13-wire / C14）——**已部分完成，剩下三块各有前置** -- **C15(b) 线路字节 + C13(wire)**：`IcebergScanRange.populateRangeParams`（现 `:330`，行号信 grep）逐 range 把完整 delete 列表 `delete.toThrift()` + partition JSON 塞进每个 `TFileRangeDesc`。真正减线路字节须 **thrift params 级 delete-file 字典 + per-range 索引 + BE reader 改** = **协议演进（FE+BE 双向 + 兼容性），非回归修复**（审计 C15 自述）——**立项前须与用户确认是否要做这个大改**。 -- **C14 fe-core 通用节点 per-split hoist**：`FileQueryScanNode`/`PluginDrivenScanNode` 逐 split `LocationPath.of`（URLEncoder+URI.create）、每 split 新建 provider（`getFileCompressType`）、造完即弃的 columns-from-path（被 `IcebergScanRange.populateRangeParams` unset）。**⚠ 框架层**、量级较小（~几百 ms@10 万 split）、须证 byte+cost 双不变。与 PERF-09 同属框架层，可一并评估。 +## PERF-11 remainder — C15b 线路字节 / C13-wire（**协议演进，碰 BE+thrift，铁律 D 必签字**） +- **病灶（HEAD 已核）**：`IcebergScanRange.populateRangeParams`（现约 `:330`，wire loop `:413-417`）逐 range `delete.toThrift()` + partition JSON 塞进**每个** `TFileRangeDesc`；一个数据文件 k 个切片各带完整 delete 列表、SDK 共享 delete 重复 M×k。BE `be/src/format/table/iceberg_reader_mixin.h:435` 逐 range inline 读 `delete_files`、无字典。 +- **成本**：线路字节 ∝ split 数 × 每分片 delete 数，删除密集/高 split MOR 表可达几 MB；仅 v2+ MOR 生效。低-中严重度。 +- **⚠ 协议演进**：真减字节须 thrift **扫描节点级 delete-file 字典 + per-range 索引**（`TFileScanRangeParams`，可仿 paimon 字段 27/30 的 scan-node-level hoist 先例）+ **BE reader 改** + **跨版本兼容矩阵**（老BE↔新FE、新BE↔老FE）。**verify 纠正**：FE 发端**不需动 fe-core**——通用 SPI 挂钩 `ConnectorScanPlanProvider.populateScanLevelParams` 已存在、paimon 已用同法挂源专属 scan-node 字段，iceberg 照抄即可、零 fe-core 加面；但 **BE reader + 兼容矩阵是大头**。FE-only「memoize/share toThrift」半赢=**零线路字节收益**（thrift 每引用仍整块重写），多半不值单做。 +- → **立项前必与用户确认是否要做这个大改**（非回归修复）。 + +## C14 剩余子项（暂缓，记录） +- #1/#2 路径重复解析（`PluginDrivenSplit.buildPath` `LocationPath.of` / `FileQueryScanNode.splitToScanRange` `toStorageLocation`）：假设"所有连接器路径已规整 + `new Path(s).toString()==s`"——Hadoop Path 规整尾斜杠/合并 `//`，**非天然字节安全**，须逐连接器证，性价比差。 +- #4 build-then-discard 分区列（`IcebergScanRange.populateRangeParams` unset 掉通用节点造的 columns-from-path，仅身份分区 iceberg 表）：干净修须新增 `ConnectorScanRange` 能力位（默认 false）→ 给 fe-core/SPI **加面**（碰铁律 A）。 +- 均低量级；用户 2026-07-19 拍板只做 #3。 ## 铁律提醒 -- PERF-11 remainder 的 C15b 碰 **BE + thrift 协议**：立项前必与用户确认（大改、非回归修复）。 -- PERF-09 与 PERF-11 的 C14 碰 **fe-core 通用节点/框架**：允许改（跨连接器普惠）但**禁按源名分支** + 须证 byte+cost 双不变 + **两段验**。 -- 立项先读审计报告 §2/§5 对应簇原始证据（findings.json 有完整调用链），再对照 HEAD 真实代码 review（HANDOFF 行号/方案可能过时）。PERF-11 已实证：C13 原证据（`IcebergRewritableDeleteStash`）PERF-07 已删——**动前必 grep 核实证据未 stale**。 +- PERF-09 碰 **fe-core 通用节点/框架**：允许改（跨连接器普惠）但**禁按源名分支** + 须证 byte+cost 双不变（本项恰恰**不能**证字节等价 → 走"接受语义变"签字路线）+ **两段验**。 +- PERF-11 的 C15b 碰 **BE + thrift 协议**：立项前必签字（铁律 D）。 +- 立项先读审计报告 §2/§5 对应簇 + findings.json 完整调用链，再对照 HEAD 真实代码 review（HANDOFF 行号/方案可能过时；本 session 已实证多处证据仍有效但行号漂移）。 --- # 🧰 构建/验证坑（**实证，务必照做**） 1. **iceberg 侧测试**：`mvn install -pl fe-connector/fe-connector-iceberg -am -Dtest='<类逗号列表>' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false [-Dcheckstyle.skip=true]`（绝对 `-f /fe/pom.xml`）。`${revision}` 未 flatten→必 `-am`；`-am test` 不产 shade jar 故用 **`install`**。 -2. **动 fe-core 者必两段验**：iceberg 连接器不依赖 fe-core，`-pl iceberg -am` 反应堆无 fe-core——fe-core 改动 + 其单测须另跑 `mvn test -pl fe-core -am -Dtest= -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -f /fe/pom.xml`。（**PERF-09 与 PERF-11 的 C14 碰 fe-core → 要两段验；纯 iceberg 项免**。）PERF-08 已实证：C19 走 fe-core 段、C21 走 iceberg 段。 +2. **动 fe-core 者两段验**：iceberg 连接器**不依赖 fe-core**，`-pl iceberg -am` 反应堆无 fe-core——fe-core 改动 + 其单测须 `mvn test -pl fe-core -am -Dtest='' -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -f /fe/pom.xml`。**反之：纯 fe-core 改动（如本次 C14-provider）iceberg 模块不受影响、无需另跑**（无编译/测试耦合）。`mvn test -pl fe-core -am` 会跑各模块 checkstyle validate（grep `@ fe-core ... 0 Checkstyle violations`）。 3. **测试必加 `-Dmaven.build.cache.enabled=false`**（否则 surefire 静默跳过）。 4. **别用 `mvn -q` 跑测试**：抑制 `BUILD SUCCESS`/`Tests run`。grep 日志 `BUILD SUCCESS` + `Tests run:`。 5. **别 `nohup ... &` 套 `run_in_background`**——maven 变孤儿、假 exit 0。直接 `mvn ... >> log 2>&1`（run_in_background）；**通知里 exit code 是 echo 的**，grep 日志 `BUILD SUCCESS`。 -6. **checkstyle 主源 ≤120**，且**扫 test 源**（删测试/改 import 后 grep 残留 unused import）。import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行。PERF-08 踩过：test helper 内联 builder 不具名类型 → 该 import 变 unused。 -7. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。工作树另有大量**与本任务无关的 untracked 杂物**（`.claude/`、`plan-doc/` 其它、`docker/`…），提交只 add 本任务具体文件。 -8. **并发探测**：本目录 `find fe -newermt '-120 sec'` + `pgrep -a -f maven`。`/mnt/disk1/yy/git/doris` 是另一 worktree，不冲突。 +6. **checkstyle 主源 ≤120**，且**扫 test 源**（删测试/改 import 后 grep 残留 unused import）。import 序 `SAME_PACKAGE(org.apache.doris.*)→THIRD_PARTY→STANDARD_JAVA` 组内字母序组间空行。 +7. **`regression-test/conf/regression-conf.groovy` 本就脏** —— 别 `git add -A`，精确 add。工作树另有大量与本任务无关的 untracked 杂物（`.claude/`、`plan-doc/` 其它、`docker/`…），提交只 add 本任务具体文件。 +8. **并发探测**：本目录 `find fe -newermt '-120 sec'`（滤 `/target/`）+ `pgrep -a -f maven`。`/mnt/disk1/yy/git/doris` 是另一 worktree，不冲突。 9. **本 worktree 的 `.git` 是文件** —— rebase 脚本用 `git rev-parse --git-path`。 --- # 🗂 遗留（已闭环，仅记录） -- **e2e 统一补**：PERF-07 的读写归一 + PERF-08 的 `rewrite_data_files`/`expire_snapshots` 计数断言 e2e，均**留 P6.6 切换阶段统一补**（当前 iceberg 未在 `SPI_READY_TYPES`，provider 休眠；对齐 `hms-iceberg-delegation-needs-e2e`）。PERF-08 的 C19 分布式实路径单测非集群不可达，亦归此。 -- 架构记忆 `iceberg-table-resolution-cache-scoping` 已随 PERF-07 落地闭环。 +- **e2e 统一补**：PERF-07 读写归一 + PERF-08 计数断言 e2e，留 P6.6 切换阶段统一补（当前 iceberg 未在 `SPI_READY_TYPES`，provider 休眠）。 +- 架构记忆 `iceberg-table-resolution-cache-scoping` 已随 PERF-07 闭环。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-scan-provider-memo-design.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-scan-provider-memo-design.md new file mode 100644 index 00000000000000..58f8c99648b839 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-scan-provider-memo-design.md @@ -0,0 +1,87 @@ +# FIX-PERF-11 (C14 slice #3) — memoize resolveScanProvider() per scan + +> **范围**：仅 C14 的第 3 个子项——**扫描 provider 每分片重复解析/分配**。C14 的其余三项（路径重复解析 #1/#2、build-then-discard 分区列 #4）**不在本次范围**（#1/#2 有跨连接器 Hadoop Path 字节风险、#4 需新增 SPI 能力位给 fe-core 加面，均按 recon 结论暂缓）。用户 2026-07-19 拍板：只做这个安全小赢。 +> 归属：`PERF-11` 的一个独立 commit（连同 C12+C15a+C13-plan 的 `10b7d29423f` 之后的第二块）。 + +## Problem + +`PluginDrivenScanNode.resolveScanProvider()`（`fe-core`，第 224-226 行）= `connector.getScanPlanProvider(currentHandle)`。SPI 契约（`Connector.java:74-83`）明确 "providers are built fresh/stateless per call"。**多数连接器每次调用 new 一个 provider 实例**（iceberg `IcebergConnector.getScanPlanProvider:670` / paimon `:283` / hive `:204`）；jdbc（`:114`）/maxcompute（`:218`）则**字段缓存**复用同一实例、es/trino 做**幂等惰性 init**——故本 memo 对前者去掉每分片重复分配、对后者是**严格 no-op**，**任何连接器都不回退**（红队跨 8 连接器逐一核实）。 + +该方法在扫描热路径被**每分片**重复调用： +- `getFileCompressType(FileSplit)`（第 603-609 行）每分片调 `resolveScanProvider()` 再 `onPluginClassLoader(...)` 做**两次类加载器切换**跑 `adjustFileCompressType`（iceberg 是 identity 空操作）。 +- `getDeleteFiles(TFileRangeDesc)`（第 673-679 行）每 range 亦调 `resolveScanProvider()`。 +- 三条枚举路径均命中：eager（`FileQueryScanNode:435` 协程内逐 split）、partition-batch（`startSplit:1551-1590` 多个 `CompletableFuture` **并发** batch 线程经 `SplitAssignment.appendBatch:114` 逐 split 回调）、streaming（`startStreamingSplit:1630` 单泵线程逐 split）。 + +**成本量级**（recon + verify 双确认）：纯本地 CPU + 短命对象分配，**无远程 IO**；每分片新建 provider（~100-200ns）+ 两次 TCCL swap。10 万 split 约几百毫秒 + GC 压力，百万 split 秒级；1 万 split 以下不可见。是审计问题类里量级最低的一档（variant D：循环不变量未 hoist、无缓存、默认配置）。 + +## Root Cause + +provider 是 `currentHandle` 的**纯函数**：SPI 契约保证 "the selected provider does not change across a scan"，单格式连接器默认忽略 handle（`Connector.java:85-86` 直接委派 no-arg），唯一按 handle 选 provider 的是 hive 异构网关（`HiveConnector.getScanPlanProvider(handle):222`，按 handle **类型/格式**选 sibling，格式在一次扫描内不变；且 hms 尚未进 `SPI_READY_TYPES`，休眠）。而 `currentHandle` 只在规划早期被 pushdown/pin 换新对象（filter `:779`/limit `:817`/projection `:836`/mvcc pin `:919`/rewrite scope `:1023`/topn `:1042`），**全部发生在 split 枚举之前**；进入逐 split 阶段后 `currentHandle` 恒定。 + +所以一次扫描内 `resolveScanProvider()` 本应至多解析**几次**（每个不同的 handle 对象一次），实际却**每分片一次**——纯重复分配。 + +**既有先例证明"共享一个 provider 实例"是安全且已在用的模式**：`startSplit:1542` / `startStreamingSplit:1628` 已在协调线程 `resolveScanProvider()` 一次，把该**单一实例**捕获进 `final scanProvider`，再在多个**并发** async 任务里共享它做 `planScanForPartitionBatch`（`:1562`）/`streamSplits`（`:1633`）。即"一次扫描共享一个 provider（含并发线程）"已是重活路径的现状。本修复只是把这份共享延伸到当前仍在每分片重解析的 `getFileCompressType`/`getDeleteFiles`。 + +## Design + +在 `PluginDrivenScanNode` 内对 `resolveScanProvider()` 做**按 `currentHandle` 身份记忆**的 memo，用**单个不可变 holder + 一次 volatile 发布**，使 partition-batch 的并发 appendBatch 线程始终读到自洽的 `(handle, provider)` 对。 + +```java +// 字段（挂在其它 cache 字段附近，如 cachedPropertiesResult 旁） +private volatile ResolvedScanProvider resolvedScanProvider; + +// 私有静态不可变 holder +private static final class ResolvedScanProvider { + private final ConnectorTableHandle handle; + private final ConnectorScanPlanProvider provider; // 可为 null（无扫描能力的连接器） + ResolvedScanProvider(ConnectorTableHandle handle, ConnectorScanPlanProvider provider) { + this.handle = handle; + this.provider = provider; + } +} + +private ConnectorScanPlanProvider resolveScanProvider() { + ResolvedScanProvider cached = resolvedScanProvider; + if (cached == null || cached.handle != currentHandle) { // 身份比较（==） + cached = new ResolvedScanProvider(currentHandle, + connector.getScanPlanProvider(currentHandle)); + resolvedScanProvider = cached; // 单次 volatile 写，原子发布 (handle, provider) + } + return cached.provider; +} +``` + +**为何这样安全且字节不变**： +- **身份键（`==`）**：每次都以当前 `currentHandle` 为键，命中才复用。`currentHandle` 一换对象（pushdown/pin）即 miss→重解析——与"每次都 `connector.getScanPlanProvider(currentHandle)`"**逐字节等价**，无需依赖"格式不随 pushdown 变"这类连接器语义推断，纯机械不变式。**无需**在 `convertPredicate` 等处显式失效（身份 miss 自动失效）。 +- **null 正确缓存**：无扫描能力连接器 provider=null，holder 携带 `(handle, null)`，下次同 handle 直接返回 null 不重解析。 +- **并发安全**：两个字段合成一个不可变 holder，经**单次 volatile 写**发布——读者要么见旧 holder、要么见新 holder，绝不会读到"新 key + 旧 provider"的撕裂组合（这正是裸双字段 memo 的数据竞争）。并发下多线程可能各建一个 holder（冗余但无害，因同一 `currentHandle`→等价 provider），每个返回值自洽。`currentHandle` 在并发阶段不再变更（pushdown/pin 都在 `CompletableFuture.runAsync` 之前），且 executor 提交建立 happens-before，async 线程读到稳定 currentHandle。 +- **共享 provider 实例安全**：本路径调用的 provider 方法（`adjustFileCompressType` iceberg=identity/hive=纯 LZ4 重映射、`getDeleteFiles` 只读 params）**皆纯函数/只读**，provider 的缓存（manifest/format/table）都挂在**长生命周期 connector 上并注入**（非 provider 自身状态）。且既有 async planScan 已并发共享一个 provider 实例（见 Root Cause 先例），本改动与现状一致。 + +**改动面**：仅 `PluginDrivenScanNode.java`——加 1 个 volatile 字段 + 1 个私有静态 holder 类 + 改 `resolveScanProvider()` 方法体（约 15 行）。12 个 `resolveScanProvider()` 调用点全部自动受益，**无一处签名变更**。 + +## Constraints check（铁律） + +- **铁律 A（fe-core 只减不增）**：本改动给 fe-core **净增** ~15 行（1 字段 + 1 holder + 方法体）。**但**：本任务空间 README 已**预先授权** PERF-09/C14 类"fe-core 通用框架层性能改动"（"允许改——惠及所有连接器、非 source-specific"），用户 2026-07-19 亦已选定本项。这是"通用框架 perf memo"，非"删旧代码期把源逻辑搬进 fe-core 的 scaffolding relocation"，属被授权的例外。**已向用户点明净增性质**。 +- **铁律 B（connector-agnostic，禁按源名分支）**：memo 纯机械（缓存一个既有 connector-agnostic 调用的结果），**零源名分支**。✓ +- **铁律 C（对所有连接器 byte+cost 双不变）**：byte——每 handle 返回等价 provider，下游 thrift 不变；cost——严格更少或相等的分配（memo），**任何连接器都不回退**。✓ +- **铁律 D（BE/thrift 协议）**：**不触碰** BE/thrift。✓ 无需该轴签字。 + +## Implementation Plan + +1. `PluginDrivenScanNode.java`：加 `private volatile ResolvedScanProvider resolvedScanProvider;` 字段（放 `cachedPropertiesResult` 附近）+ 私有静态 `ResolvedScanProvider` holder 类（放文件内既有私有静态类约定位置）+ 改 `resolveScanProvider()` 方法体为上述 memo。方法 javadoc 补一句"memoized per currentHandle"。 +2. 不改任何调用点、不改任何 SPI、不改任何连接器。 + +## Risk + +- **低**。唯一微妙点是并发（partition-batch），已由不可变 holder + volatile 发布消解，且与既有 async 共享 provider 的现状一致。 +- 反向验证锚点：既有测试 `PluginDrivenScanNodeScanProviderSelectionTest.resolvesProviderForCurrentHandle` 断言"handle 变则重解析、不永久缓存首个"——本 memo 身份键**保住**该行为（handle 变→miss→重解析），故该测试是天然的 parity 守门，必须继续绿。 + +## Test Plan（含度量守门） + +- **新守门测试**（扩 `PluginDrivenScanNodeScanProviderSelectionTest`）：`memoizesProviderForStableHandleAndReResolvesOnHandleChange`——用计数 Mock connector： + - 固定 `currentHandle=icebergHandle`，调 `resolveScanProvider()` 3 次 → `Mockito.verify(connector, times(1)).getScanPlanProvider(icebergHandle)`（证每分片重分配已消除，修前会是 3 次）。 + - 换 `currentHandle=hiveHandle`，调 2 次 → `verify(connector, times(1)).getScanPlanProvider(hiveHandle)` + `assertSame(hiveProvider, ...)`（证 handle 变则重解析、且再命中缓存）。 + - 镜像既有测试的 `Mockito.CALLS_REAL_METHODS` 部分 mock + `Deencapsulation.setField/invoke` 构造法（对齐 `catalog-spi-fe-core-test-infra` 记忆）。 +- **既有测试全绿**（byte-parity）：`PluginDrivenScanNodeScanProviderSelectionTest` + `PluginDrivenScanNode*Test`（DeleteFiles/ClassifyColumn/BatchMode/... 全套）+ 相关连接器 UT 不需改一行。 +- **两段验**（本项碰 fe-core）：iceberg 连接器不依赖 fe-core，故 fe-core 改动 + 其单测须**单独** `mvn test -pl fe-core -am -Dtest=PluginDrivenScanNode*Test -DfailIfNoTests=false -Dmaven.build.cache.enabled=false -f /fe/pom.xml`；另跑一遍 iceberg 全模块 `install` 确认无回归。 +- **度量口径**：守门测试的 `verify(times(1))` 即"每 handle 解析 1 次"的直接断言（对齐 PERF-01/03 的 `loadCountForTest()==1` 模式）；并发正确性由构造保证（volatile 不可变 holder），不写 flaky 压测。 diff --git a/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-scan-provider-memo-summary.md b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-scan-provider-memo-summary.md new file mode 100644 index 00000000000000..f89a9f24058d03 --- /dev/null +++ b/plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-11-scan-provider-memo-summary.md @@ -0,0 +1,44 @@ +# FIX-PERF-11 (C14 slice #3) — 小结：memoize resolveScanProvider() per scan + +> 设计见 [`FIX-PERF-11-scan-provider-memo-design.md`](./FIX-PERF-11-scan-provider-memo-design.md)。仅 C14 第 3 子项（provider 每分片重复分配），其余 C14 子项按 recon 结论暂缓（用户 2026-07-19 定）。 + +## Problem + +`PluginDrivenScanNode.resolveScanProvider()`（fe-core 通用扫描节点）= `connector.getScanPlanProvider(currentHandle)`，SPI 契约"providers are built fresh/stateless per call"。它在**每分片/每 range** 热路径被重复调用（`getFileCompressType:603` 每 split、`getDeleteFiles:673` 每 range，另 ~10 个规划调用点），对 new-per-call 连接器（iceberg/paimon/hive）每分片**重新分配一个 provider + 两次 TCCL 类加载器切换**（对 iceberg 是 identity 空操作）。三条枚举路径全中：eager 协调线程逐 split、partition-batch 多并发 batch 线程、streaming 单泵线程。量级为审计问题类最低档（~100-200ns/split、无远程 IO，10 万 split 约几百毫秒 + GC 压力）。 + +## Root Cause + +provider 是 `currentHandle` 的纯函数（单格式连接器忽略 handle；唯一按 handle 选的 hive 网关按**格式**选、格式一次扫描内不变，且 hms 休眠）。`currentHandle` 只在规划早期 pushdown/pin 换新对象，进入 split 枚举后恒定 → 一次扫描本应至多解析几次，实际每分片一次。且"整次扫描共享一个 provider 实例"**已是**重活路径（`startSplit:1542`/`startStreamingSplit:1628` 捕获一次、并发 async planScan 共享）的现状。 + +## Fix + +对 `resolveScanProvider()` 做**按 `currentHandle` 身份键**的 memo：单个不可变 `ResolvedScanProvider{handle, provider}` holder，经**一次 volatile 写**发布。命中（`holder.handle == currentHandle`）复用；miss（首次或 handle 被 pushdown/pin 换新对象）重解析并重发布。null provider（无扫描能力连接器）正确缓存返回。 + +- **逐字节不变**：身份键使其与"每次都 `connector.getScanPlanProvider(currentHandle)`"机械等价，不依赖任何连接器语义假设；下游 thrift 不变。 +- **并发安全**：final 字段 + 单次 volatile 写 = 安全发布，并发 appendBatch 线程绝不读到"新键配旧值"的撕裂；冗余并发构建无害（同 handle→等价 provider）。与既有 async 共享 provider 现状一致。 +- **改动面**：仅 `PluginDrivenScanNode.java`——1 个 volatile 字段 + 1 个私有静态 holder 类 + 改 `resolveScanProvider()` 方法体，约 15 行。零调用点/SPI/连接器改动。 + +## 铁律核对 + +- A（fe-core 只减不增）：净增 ~15 行；本任务空间 README 预授权"通用框架层 perf 改动"，用户 2026-07-19 选定，已向用户明示净增。属被授权例外（非 scaffolding relocation）。 +- B（禁按源名分支）：纯机械 memo，零源名分支。✓ +- C（对所有连接器 byte+cost 双不变）：byte 等价；cost 严格更少或相等（memo），无连接器回退。✓ +- D（BE/thrift）：不触碰。✓ + +## 设计红队（3 视角对抗，clean-room） + +并发/JMM、跨 8 连接器字节等价、生命周期与必要性——**三者全 SURVIVES、0 缺陷、一致"照方案实现"**。红队额外核实：所有连接器的 `getScanPlanProvider` 要么 new-per-call（本改动省分配）、要么已字段缓存（jdbc/maxcompute，本改动为严格 no-op），无一回退；provider 实例除 iceberg `scanProfileStash`（仅经 getSplits 单一捕获 local 走、不经 per-split resolve）外皆 final/不可变；per-split 方法（`adjustFileCompressType`/`getDeleteFiles`/`classifyColumn`）皆纯函数。修正设计文档一处笔误（jdbc/maxcompute 实为字段缓存非 new-per-call，方向安全）。 + +## Tests + +- **TDD 守门（新）**：扩 `PluginDrivenScanNodeScanProviderSelectionTest.memoizesProviderForStableHandleAndReResolvesOnHandleChange`——固定 handle 连调 3 次断言底层 `getScanPlanProvider` **恰 1 次**（修前 3 次，先观察 RED：`Wanted 1 time ... but was 3` 确证）；换 handle 后再 2 次断言重解析恰 1 次 + provider 匹配。镜像既有测试的 `CALLS_REAL_METHODS`+`Deencapsulation` 构造。 +- **既有 parity 全绿**：`PluginDrivenScanNode*Test` 全 19 类 **103 pass / 0 fail / 0 error / 0 skip**（含天然守门 `resolvesProviderForCurrentHandle`：换 handle 须重解析）。连接器单测不需改一行。 +- **两段验（fe-core）**：`mvn test -pl fe-core -am -Dtest='PluginDrivenScanNode*Test'` **BUILD SUCCESS + fe-core Checkstyle 0 违规**。change 为 fe-core-only、iceberg 连接器不依赖 fe-core → iceberg 模块不受影响（无需另跑，无编译/测试耦合）。 + +## Result + +`87ff73b1a95`：fe-core `PluginDrivenScanNode` scan-provider 每分片重分配 O(splits)→ 每 handle 1 次。度量守门 `verify(times(1))`。全 `PluginDrivenScanNode*Test` 103 绿 + 0 checkstyle，0 回归。 + +## 度量口径 + +守门测试 `verify(connector, times(1)).getScanPlanProvider(handle)` 直接断言"每 handle 解析 1 次"（对齐 PERF-01/03 `loadCountForTest()==1`）；并发正确性由构造（不可变 holder + volatile）保证，不写 flaky 压测。 diff --git a/plan-doc/perf-hotpath-iceberg/progress.md b/plan-doc/perf-hotpath-iceberg/progress.md index 86fd3ec7fc59c5..988233a65a89f7 100644 --- a/plan-doc/perf-hotpath-iceberg/progress.md +++ b/plan-doc/perf-hotpath-iceberg/progress.md @@ -143,3 +143,19 @@ - **明确不做**（tasklist 记）：**C15(b) 线路字节**（`populateRangeParams` 逐 range `toThrift()`+partition JSON 塞每个 `TFileRangeDesc`，减字节须 thrift 字典+BE reader 改=协议演进非回归修复）、**C13(wire) 二次转换**（同处线路侧、随 C15b）、**C14 fe-core 通用节点 hoist**（框架层、须双不变）。 - **结果**：`IcebergScanPlanProviderTest` 105/105（+2）、`IcebergConnectorTransactionTest` 66/66、**全模块 1064 pass/1 skip/0 fail**、BUILD SUCCESS + 0 checkstyle。 - **下一步**：见 HANDOFF —— PERF-09（fe-core streaming pump 微批）或 PERF-11 remainder（C15b/C13-wire/C14）。 + +--- + +## 2026-07-19 — session 4：P2 剩余复核（3 候选）+ PERF-11 C14-provider-memo 落地（commit `87ff73b1a95`） + +- **开场按 README step 4 复核 3 个剩余候选并交用户定**（HEAD 55087e08d0c，行号信 grep）：派 3 recon + 3 对抗 verify agent，全判**证据未失效**、3 verify 全 CONFIRMED。要点： + - **streaming pump 微批（C5/PERF-09）**：`PluginDrivenScanNode.startStreamingSplit:1638-1642` 逐 split `addToQueue`→`FederationBackendPolicy.computeScanRangeAssignment` 每 split 重建（audit 还**漏算** `equateDistribution` O(B log B)+两堆恒跑、`ResettableRandomizedIterator` 拷贝在默认 ROUND_ROBIN 下死重）。纯 CPU/GC（10⁵ split≈0.1-0.5s），触 iceberg+trino 两流式。**非字节等价**：攒批令 shuffle+均衡真生效→**后端分配结果会变**（功能安全、纯负载均衡启发式变），且是上游基线演进非回归修——**需用户签"接受语义变"**。 + - **C14 通用节点（PERF-11 剩）**：4 子项全在 HEAD。**#3 provider memo=安全小赢**（纯 fe-core、近零字节风险）；#1/#2 路径少解析有跨连接器 Hadoop Path 字节风险；#4 build-then-discard 须新增 SPI 能力位（碰铁律 A）。byte-identical thrift、iron D 不触。 + - **C15b/C13-wire（PERF-11 剩）**：`IcebergScanRange.populateRangeParams:413-417` 逐 range `delete.toThrift()`+partition JSON 塞每个 `TFileRangeDesc`。真减线路字节=thrift 扫描节点级删除字典+per-range 索引+**BE reader 改**+跨版本兼容=**协议演进（iron D，需签字）**。verify 纠正 recon 一处：FE 发端**不需动 fe-core**（`populateScanLevelParams` SPI 挂钩已存在、paimon 已用），但 BE+兼容矩阵是大头。FE-only「memoize toThrift」半赢**零线路字节收益**。 +- **用户拍板**：只做 **C14 #3 provider memo**（干净低风险小赢），其余暂缓。 +- **设计 + 红队（clean-room，3 视角对抗）**：并发/JMM · 跨 8 连接器字节等价 · 生命周期与必要性——**全 SURVIVES、0 缺陷、一致"照做"**。关键确认：provider 是 `currentHandle` 纯函数（单格式忽略 handle、hive 网关按格式选、休眠）；`currentHandle` 只规划早期 pushdown/pin 换新对象、split 枚举后恒定；**"整次扫描共享一个 provider" 已是现状**（`startSplit:1542`/`startStreamingSplit:1628` 捕获一次、并发 async planScan 共享）；per-split 方法（`adjustFileCompressType`/`getDeleteFiles`/`classifyColumn`）纯函数；jdbc/maxcompute 已字段缓存→memo 对其 no-op。修正设计一处笔误（那两连接器非 new-per-call，方向安全）。 +- **落地（TDD，fe-core 一个 `[perf]` commit）**：`resolveScanProvider()` 改按 `currentHandle` **身份键** memo——不可变 `ResolvedScanProvider{handle,provider}` holder + **单次 volatile 写**发布（并发 partition-batch appendBatch 线程读自洽对、禁撕裂）；命中复用、身份 miss 重解析、null provider 正确缓存。仅 `PluginDrivenScanNode.java` +1 volatile 字段 +1 私有静态 holder 类 + 方法体 ~15 行；零调用点/SPI/连接器改动。 +- **先 RED 后 GREEN**：新守门 `PluginDrivenScanNodeScanProviderSelectionTest.memoizesProviderForStableHandleAndReResolvesOnHandleChange` 在旧码 `Wanted 1 time ... but was 3` 确证 RED（每 split 重解析）；实现后固定 handle 连调 3 次 `verify(times(1))`、换 handle 再 2 次各 1 次 + provider 匹配。 +- **结果**：全 `PluginDrivenScanNode*Test` 19 类 **103 pass / 0 fail / 0 skip**、**fe-core Checkstyle 0 违规**、BUILD SUCCESS、0 回归。度量：per-split provider 重分配 O(splits)→ 每 handle 1 次。**fe-core-only、iceberg 不依赖 fe-core → 无编译/测试耦合、不受影响**（无需另跑 iceberg 模块）。 +- **新判据（可复用）**：「共享框架热路径的 memo：以句柄/键的**身份（==）**为键即与逐次调用**机械等价**、免依赖连接器语义假设；并发下用**不可变 holder + 单次 volatile 写**安全发布，且先确认该共享是否已是现状（现状即许可）」。 +- **下一步**：见 HANDOFF —— P2 剩余仅存 PERF-09（流式 pump 微批，需签"接受分配语义变"）与 PERF-11 remainder 的 C15b/C13-wire（协议演进，需签字）+ C14 #1/#2/#4（暂缓），均已在本 session 复核清楚、待用户择时定夺或收尾。 diff --git a/plan-doc/perf-hotpath-iceberg/tasklist.md b/plan-doc/perf-hotpath-iceberg/tasklist.md index f8107aa0e20ef3..a349f0aec5b0b2 100644 --- a/plan-doc/perf-hotpath-iceberg/tasklist.md +++ b/plan-doc/perf-hotpath-iceberg/tasklist.md @@ -25,7 +25,7 @@ | PERF-08 | P2 | C19 C21 | 维护路径逐单位重扫 / 无去重（rewrite_data_files 每 group planFiles；expire_snapshots S×M）→ union 一次注册 / 按 path 去重 | 改动小可先行 | ✅ 完成 | `89cc39c8d88`(C19)+`be0035eff62`(C21) | | PERF-09 | P2 | C5 | **（fe-core 框架层）** streaming pump 逐 split 重建 backend 候选集 + 锁往返 → 微批 | ⚠ 跨连接器，见约束 | ⏳ | | | PERF-10 | P2 | C8 | 同组 WHERE conjunct 被转换 5~6 次/查询 → node 字段 memo（与 01 同点失效） | 与 01 同源 | 🔬 暂缓（复核判低价值） | | -| PERF-11 | P2 | C12 C13 C14 C15 | per-split 不变量重算 + payload 逐 slice 复制 → hoist / (specId,PartitionData) memo / per-file 共享 | 同区域批量 · ⚠ 含 fe-core 通用节点 | 🚧 部分完成（C12+C15a+C13-plan） | `10b7d29423f` | +| PERF-11 | P2 | C12 C13 C14 C15 | per-split 不变量重算 + payload 逐 slice 复制 → hoist / (specId,PartitionData) memo / per-file 共享 + 通用节点 provider memo | 同区域批量 · ⚠ 含 fe-core 通用节点 | 🚧 部分完成（C12+C15a+C13-plan+C14-provider） | `10b7d29423f`+`87ff73b1a95`(C14-provider) | --- @@ -104,13 +104,13 @@ - **处置**：用户 2026-07-18 拍板**跳过**，改做高价值 PERF-11。若将来愿加一个中性 `ConnectorSession` explain 信号，连接器侧 explain-gate 可复活（收益仍微秒级）。 - **依赖**：与 PERF-01 同源，宜在 01 之后顺手。 -### [🚧] PERF-11 — 簇5：per-split 不变量 + payload 放大（C12 C13 C14 C15） · 部分完成 `10b7d29423f` +### [🚧] PERF-11 — 簇5：per-split 不变量 + payload 放大（C12 C13 C14 C15） · 部分完成 `10b7d29423f`+`87ff73b1a95` > 同一 `buildRange`/`populateRangeParams` 区域的 per-split 重复计算与 payload 放大，拆多 commit。权威设计/小结/红队见 [`designs/FIX-PERF-11-per-file-invariant-memo-*`](./designs/FIX-PERF-11-per-file-invariant-memo-design.md)。 > **已完成（`10b7d29423f`，连接器侧）= C12 + C15(a) + C13(plan 侧)**：`buildRange` 穿 1 条目 `PerFileScratch`（键 `task.file()` 身份），partition JSON/identity/ordered 值/delete 载体 per-slice→per-file；同文件 k range 共享不可变实例（省 FE heap）；v3 `rewritableDeleteSupply.put` per-slice→per-file（文件首 slice、覆盖末文件）。eager+流式共用 `buildRangeForTask` choke point，流式 scratch O(1) 不破 OOM。3 视角对抗 byte-parity 0 破坏；全模块 1064 pass/1 skip。 > **仍未做（deferred，另立 commit/任务）**： > - **C15(b) 线路字节**：`populateRangeParams` 逐 range 把完整 delete 列表 `toThrift()` + partition JSON 塞进每个 `TFileRangeDesc` —— 真正减线路字节须 thrift params 级 delete-file 字典 + per-range 索引 + **BE reader 改**，是**协议演进非回归修复**（审计 C15 自述）。 > - **C13(wire) 二次转换**：上面的 `populateRangeParams` per-range `delete.toThrift()` 亦是 C13 的「第二次转换」——与 C15b 同一处线路侧，随 C15b 一并处理或单独 hoist（缓存 per-file 已转换 descs，牵扯持有 thrift 对象的内存权衡）。 -> - **C14 fe-core 通用节点 hoist**：见下（框架层，须双不变，另议）。 +> - **C14 fe-core 通用节点**：✅ **provider memo（子项 #3）已完成** `87ff73b1a95`——`PluginDrivenScanNode.resolveScanProvider` per-split 重分配 provider → 按 `currentHandle` 身份 memo（不可变 holder + volatile 发布），byte 等价、3 视角红队 0 缺陷；权威设计/小结见 [`designs/FIX-PERF-11-scan-provider-memo-*`](./designs/FIX-PERF-11-scan-provider-memo-design.md)。**剩余暂缓**（用户 2026-07-19 拍板只做 #3）：路径重复解析（#1/#2，`LocationPath.of`/`toStorageLocation` 有跨连接器 Hadoop Path 规整字节风险、性价比差）、build-then-discard 分区列（#4，须新增 `ConnectorScanRange` 能力位给 fe-core 加面、碰铁律 A）。 - **C12**：一个 data file 被 `TableScanUtil.splitFiles` 切成 k 个 byte-slice 后，`buildRange:1045` 对每个 slice 重算 partition JSON（Jackson 序列化 + 时区格式化）、identity map、delete 转换 —— (specId, PartitionData) 级不变量。100k split ≈ 0.5~2s CPU。修：按 (specId, PartitionData) memo。 - **C13**：v3 rewritable-delete stash：同一 delete 列表 plan 期 `rewritableDeleteDescs:302-313` 转一次 thrift，`populateRangeParams` 再转一次；每 slice 重复 put 相同 supply。修：转一次复用、per-file 而非 per-slice。 - **C14**：通用节点 per-split 重复 `LocationPath.of`（URLEncoder+URI.create）、重建 columns-from-path 却被 `IcebergScanRange.populateRangeParams:435-437` unset 丢弃；`resolveScanProvider` 每 split 经 `getFileCompressType` 反复解析。修：hoist 不变量 / 删造完即弃分支。⚠ 涉 **fe-core 通用节点**——保持 connector-agnostic，勿加 source-specific 分支。 From 96c36aa0532e4d91af0fb56761ebdf9259742f09 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 09:12:31 +0800 Subject: [PATCH 27/47] [doc](catalog) add master-todo + per-statement-table-owner-port tracking spaces - plan-doc/master-todo/: a cross-task register of deferred, decision-gated big items (needs user sign-off or protocol evolution), so they are not forgotten. Seeds two entries with full background/current-call-stack/solution-call-stack/example writeups: the streaming split-dispatch micro-batch (fe-core; not byte-identical -> changes backend split distribution) and the wire-side delete-file dictionary (thrift + BE reader + mixed-version compat). - plan-doc/per-statement-table-owner-port/: tracking space for porting iceberg's PERF-07 per-statement table-load owner pattern to other connectors. Key finding recorded: the neutral fe-core/fe-connector-api scaffolding (ConnectorStatementScope + ConnectorSession.getStatementScope + StatementContext plumbing) is already in place, so porting is connector-side-only work. Candidate map (to be scoped next session): paimon (high) / hive-hms (mid-high) / hudi (mid); maxcompute/es/jdbc/trino likely excluded (no per-statement metastore loadTable fan-out). No product code touched. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- plan-doc/master-todo/README.md | 29 +++++ .../streaming-split-dispatch-microbatch.md | 92 ++++++++++++++ .../wire-delete-file-dictionary.md | 116 ++++++++++++++++++ .../per-statement-table-owner-port/HANDOFF.md | 36 ++++++ .../per-statement-table-owner-port/README.md | 88 +++++++++++++ .../progress.md | 13 ++ .../tasklist.md | 25 ++++ 7 files changed, 399 insertions(+) create mode 100644 plan-doc/master-todo/README.md create mode 100644 plan-doc/master-todo/streaming-split-dispatch-microbatch.md create mode 100644 plan-doc/master-todo/wire-delete-file-dictionary.md create mode 100644 plan-doc/per-statement-table-owner-port/HANDOFF.md create mode 100644 plan-doc/per-statement-table-owner-port/README.md create mode 100644 plan-doc/per-statement-table-owner-port/progress.md create mode 100644 plan-doc/per-statement-table-owner-port/tasklist.md diff --git a/plan-doc/master-todo/README.md b/plan-doc/master-todo/README.md new file mode 100644 index 00000000000000..af9b27f61fd112 --- /dev/null +++ b/plan-doc/master-todo/README.md @@ -0,0 +1,29 @@ +# 📌 Master TODO —— 跨任务 · 需拍板 / 协议演进级的待决大项 + +> **用途**：集中记录那些**已被复核清楚、但暂缓推进**的大项——它们不是普通的"待修 bug"，而是需要**用户先拍板**（改变行为语义、或涉及协议演进）、或跨多个任务边界的工程决策。放在这里是为了**不被遗忘**：随时可以拿出来重新评估、立项。 +> +> **收录标准**（满足其一即入）： +> 1. 修复会**改变可观察行为/语义**（哪怕功能安全），须用户显式接受； +> 2. 涉及**协议演进**（改 thrift + 改 BE + 跨版本兼容），非纯 FE 回归修复； +> 3. 跨多个任务空间、或收益/代价需要人来权衡后才决定做不做。 +> +> **不收录**：普通可直接做的回归修复 / 性能修复（那些走各自任务空间的 task-list + git）。 +> +> **每项一个独立 md**，含：背景 · 当前代码调用栈与问题 · 方案调用栈与问题 · 示例 · 状态 / 为何需拍板。行号信 HEAD、以 `grep` 为准（文档行号会随代码漂移）。 + +--- + +## 索引 + +| # | 待决项 | 改动层 | 收益 | 为何需拍板 | 状态 | 文档 | +|---|---|---|---|---|---|---| +| 1 | 流式扫描「分片派发」逐分片重建后端分配集 → 微批 | 仅 fe-core 通用框架 | 省流式大扫描规划 CPU（10⁵ 分片≈0.1~0.5s） | **非字节等价**——微批会改变分片落到哪些后端节点（有意的负载均衡语义变更） | ⏳ 待定 | [`streaming-split-dispatch-microbatch.md`](./streaming-split-dispatch-microbatch.md) | +| 2 | 线路上删除文件列表逐分片重复 → 扫描节点级删除文件字典 + 每范围索引 | FE + **thrift 协议 + BE 读取端** | 减发给 BE 的计划体积（删除密集 MOR 表可省几 MB） | **协议演进**——改 BE + 跨版本兼容矩阵，非回归修复 | ⏳ 待定 | [`wire-delete-file-dictionary.md`](./wire-delete-file-dictionary.md) | + +--- + +## 说明 + +- 这两项来自 iceberg 连接器热路径重操作审计的收尾复核（2026-07-19，证据已在 HEAD 上重新核实、均未失效）。同批次的其余低量级项（通用节点路径重复解析 / build-then-discard 分区列）性价比低或碰"fe-core 只减不增"铁律，已暂缓，未单独入册。 +- 若将来推进任一项，按项目惯例走：**先设计 → 用户确认 → 再动代码**；协议演进项（第 2 项）**动 BE/thrift 前必须签字**。 +- 内部关联（审计原始证据 + 逐项调用链）：`plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17*` 与任务空间 `plan-doc/perf-hotpath-iceberg/`。 diff --git a/plan-doc/master-todo/streaming-split-dispatch-microbatch.md b/plan-doc/master-todo/streaming-split-dispatch-microbatch.md new file mode 100644 index 00000000000000..1897cb3a4d1666 --- /dev/null +++ b/plan-doc/master-todo/streaming-split-dispatch-microbatch.md @@ -0,0 +1,92 @@ +# 待决项 1 —— 流式扫描的「分片派发」每次只送一个分片给后端分配器 + +> **状态**：⏳ 待定（**需用户签"接受后端分配语义变化"**才能立项） +> **改动层**：仅 fe-core 通用框架（`PluginDrivenScanNode` / `SplitAssignment` / `FederationBackendPolicy`），不触 BE、不触 thrift。 +> **影响面**：流式路径由 **iceberg 和 trino 两个连接器**共用——通用改动同时影响两者（不能只改 iceberg，否则违反"通用节点禁按源名分支"）。 +> **行号信 HEAD（55087e08d0c 附近，PERF-11 后）**，以 `grep` 为准。 + +--- + +## 背景 + +当一张外表非常大（匹配文件数 ≥ `num_files_in_batch_mode`，默认 1024；实际针对 10 万~100 万分片的扫描），扫描走**流式**模式：不一次性把所有分片算出来堆在 FE 内存里（会 OOM），而是一边惰性读元数据、一边把分片"泵"给调度器，靠背压（队列满就等）把 FE 堆压住。 + +问题不在"读元数据"（那是必要成本），而在**把分片交给"后端分配器"的方式**——分配器（`FederationBackendPolicy`）负责决定"这个分片让哪台 BE 去扫"。 + +--- + +## 当前代码的调用栈与问题 + +``` +startStreamingSplit() PluginDrivenScanNode.java:1600 [单个后台任务] +└─ while (needMoreSplit() && source.hasNext()): :1638 ← 逐个分片的泵循环 + one = [ new PluginDrivenSplit(source.next()) ] :1639-1640 ← 每次只包 1 个分片的 List + splitAssignment.addToQueue(one) :1641 → SplitAssignment.java:143 + └─ synchronized (assignLock) { ... } SplitAssignment.java:148 ← 每分片一次加锁往返 + backendPolicy.computeScanRangeAssignment(one) :153 → FederationBackendPolicy.java:225 + ├─ Collections.shuffle(one, seeded) :228 ← 对 1 个元素洗牌 = 空操作 + ├─ backends = flatten(backendMap) :231-234 ← 把全部 ~100 台 BE 拷进新 List + ├─ new ResettableRandomizedIterator(backends) :235 ← 又拷一遍(默认 ROUND_ROBIN 用不到) + ├─ 选 1 台 BE (ROUND_ROBIN: nextBe++) :269-270 + └─ if (enableSplitsRedistribution) :307 ← 默认 true、无生产关闭途径 + equateDistribution(assignment) :320 + ├─ allNodes 排序 O(B·logB) :329 + └─ 建两个 IndexedPriorityQueue :335-343 ← 对全部 BE 各建一遍堆 +``` + +**问题**：泵循环每吐一个分片，`computeScanRangeAssignment` 就把"和这一个分片无关的固定开销"从头重算一遍——把上百台 BE 拷两遍、建 multimap、洗牌、加锁往返，**还有 `equateDistribution`**：它对全部后端做一次 `O(B·logB)` 排序 + 建两个堆，而且默认恒开（`enableSplitsRedistribution=true`，那个 setter 只有测试在调、生产没有关它的路径）。 + +100 万分片 × 上百台 BE，这些"每分片重建"的操作叠起来是纯 FE 的 CPU + 大量短命对象（GC 压力）。**没有远程 IO**，所以量级有限（10 万分片约 0.1~0.5 秒、100 万分片几秒），但确实是白干——这些固定开销本该一批分片摊一次。 + +> 对照：分区批风格的兄弟路径已经是"整批一起 `addToQueue`"（`PluginDrivenScanNode.java:1570` 附近），非流式的 legacy 路径也是"所有分片一次 `computeScanRangeAssignment`"（`FileQueryScanNode.java:431`）——唯独流式泵是"一次一个"。 + +--- + +## 解决方案的调用栈与问题 + +方向：泵侧**微批**——攒够 K 个（64~256）再一起送。 + +``` +startStreamingSplit() +└─ batch = new ArrayList(K) + while (needMoreSplit() && source.hasNext()): + batch.add(new PluginDrivenSplit(source.next())) + if (batch.size() >= K) { + splitAssignment.addToQueue(batch) ← 一次送 K 个 + batch = new ArrayList(K) + } + if (!batch.isEmpty()) splitAssignment.addToQueue(batch) ← 末批必须 flush，否则丢分片 + splitAssignment.finishSchedule() + └─ computeScanRangeAssignment(batch) ← 从"每分片一次"变"每 K 个一次" +``` + +**方案自身的问题（正是它需要用户签字的原因）**： + +1. **不是字节等价——后端分配结果会变**（关键）。当前"一次一个"下，`Collections.shuffle`（对单元素是空操作）和 `equateDistribution`（单分片没什么可再平衡的）实际都不起作用；一旦变成"一批 K 个"，这两步就**开始真正做事**：洗牌重排这 K 个的顺序、`equateDistribution` 在这一批内部把分片从"堆得多的 BE"挪到"堆得少的 BE"。而 `nextBe`、`assignedWeightPerBackend` 是**跨批累积的实例字段**（`FederationBackendPolicy.java:84/88`）。所以**同一批分片最终落到哪些 BE 会和现在不同**。 + - 功能上安全：每个分片仍恰好被分配一次、所有数据都读到，变的只是"哪个分片去哪台机器"——负载均衡的启发式结果（甚至更均衡）。 + - 但它**违反"共享框架热路径须逐字节不变"**纪律，不能声称"透明无感"。且"一次一个"本身是照搬**上游** legacy `doStartSplit`（`:1635-1637` 注释写明），改它属于"在上游基线上演进"，不是"修回归"。→ **需要用户签"接受这个负载均衡分配变化"**。 + +2. **正确性不变式要小心**：`needMoreSplit()` 背压要**每批**重新检查（不能攒到一半该停了还继续攒）；末批一定要 flush；批边界不能丢/重分片；`source.close()` 仍在 finally 里吞异常。这些可单测。 + +3. **验证**：碰 fe-core → 两段验（iceberg 连接器不依赖 fe-core）。但"省了多少时延"和"分配分布变成什么样"这两个承载性结论**单测测不出来**，要真 BE 分布式跑 ≥1024 文件的流式扫描（iceberg + trino 各一次）才能观测；正确性不变式（不丢不重/背压）可用 mock split source 单测。 + +--- + +## 示例 + +设 100 台 BE、一次流式扫描 50 万分片、微批 K=128： +- **当前**：`computeScanRangeAssignment` 被调 **50 万次**，每次拷 100 台 BE 两遍 + 排序 100·log100 + 建两个堆 + 一次加锁 → 约 50 万次全套固定开销。 +- **微批后**：被调 **≈3900 次**（50 万 / 128），固定开销摊薄到 1/128。 +- **代价示例**：假设分片按 round-robin，当前第 7 个分片去 BE#7、第 8 个去 BE#8……微批后，这 128 个先被洗牌重排、再被 `equateDistribution` 按累积权重挪动，**第 7 个分片可能落到 BE#42**。查询结果完全一样，但"哪台机器扫哪个文件"的分布图变了——这就是需要用户点头的那个"语义变化"。 + +--- + +## 立项前须确认 + +- [ ] 用户签字：**接受微批带来的后端分片分配分布变化**（功能等价、负载均衡启发式变）。 +- [ ] 定 K（批大小）与背压交互；确认微批不改"每分片恰分一次 / 不丢不重"。 +- [ ] 两段验 + 真 BE 分布式 smoke（iceberg + trino 流式各一次）。 + +## 关联 + +审计原始证据 / 逐项调用链：`plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17*`（该簇 findings）；任务空间 `plan-doc/perf-hotpath-iceberg/`。 diff --git a/plan-doc/master-todo/wire-delete-file-dictionary.md b/plan-doc/master-todo/wire-delete-file-dictionary.md new file mode 100644 index 00000000000000..cf6962a837c06b --- /dev/null +++ b/plan-doc/master-todo/wire-delete-file-dictionary.md @@ -0,0 +1,116 @@ +# 待决项 2 —— 线路上「删除文件列表 + 分区 JSON」被逐分片重复塞进每个范围 + +> **状态**：⏳ 待定（**协议演进：动 BE/thrift 前必须用户签字**） +> **改动层**：FE 发送端（iceberg 连接器侧）+ **thrift 协议（新字段）** + **BE 读取端** + **跨版本兼容矩阵**。 +> **触碰铁律 D**（改 BE + 改协议 = 协议演进，非回归修复）。 +> **仅对 v2+ MOR 表生效**，普通表零影响。 +> **行号信 HEAD（55087e08d0c 附近，PERF-11 后）**，以 `grep` 为准。 + +--- + +## 背景 + +Iceberg 的 v2/v3 表支持"读时合并"（MOR）：数据文件旁边挂着**删除文件**（position delete / equality delete / deletion vector），BE 读数据时用它们过滤掉被删的行。FE 规划时把每个数据文件的删除文件列表放进发给 BE 的扫描计划。 + +两个放大点叠加： +- 一个大数据文件会被切成 **k 个字节切片**（每个切片是一个扫描范围 `TFileRangeDesc`），BE 并行读； +- 一个 equality delete 文件常被**很多数据文件共享**（它是分区级/表级的删除）。 + +> 上一步的提交（PERF-11 的 `10b7d29423f`）只优化了 **FE 内存侧**——让同一文件的 k 个切片在 FE 堆里**共享**同一份删除列表对象；**线路（发给 BE 的字节）上的重复没动**，就是本项。 + +--- + +## 当前代码的调用栈与问题 + +``` +createFileRangeDesc(...) FileScanNode (每个切片一次) +└─ setScanParams(rangeDesc, split) PluginDrivenScanNode.java:1693 [每范围] + └─ scanRange.populateRangeParams(fmtDesc, rangeDesc) :1705 → IcebergScanRange.java:330 + fileDesc.setPartitionDataJson(partitionJson) :397 ← 分区 JSON 塞进本范围 + if (v2+): :408 + deleteDescs = new ArrayList(deleteFiles.size()) :413 + for (delete : deleteFiles): :414 ← 每范围 × 每删除文件 + deleteDescs.add(delete.toThrift()) :415 ← 每次新建一个 TIcebergDeleteFileDesc + fileDesc.setDeleteFiles(deleteDescs) :417 ← 整份删除列表塞进"本切片"的范围 + rangeDesc.setTableFormatParams(fmtDesc) :1707 +``` + +thrift 的形状（`gensrc/thrift/PlanNodes.thrift`）： +``` +TFileRangeDesc (每范围一个) → table_format_params: TTableFormatFileDesc :565 / :464 +TTableFormatFileDesc → iceberg_params: TIcebergFileDesc +TIcebergFileDesc.delete_files : list :339 ← 删除列表挂在"每范围"里 +TIcebergDeleteFileDesc { path; bounds; field_ids; content; DV偏移; original_path; ... } :315-331 + ← 每条约 100~200 字节(主要是路径字符串) +``` + +BE 侧逐范围 inline 消费（无字典）： +``` +IcebergReaderMixin ... 初始化删除文件 be/src/format/table/iceberg_reader_mixin.h:435 + table_desc = get_scan_range().table_format_params.iceberg_params [每范围] + for (desc : table_desc.delete_files): :444 ← 逐范围内联遍历 + 分桶到 position / equality / deletion_vector :445-451 +``` + +**问题**：一个数据文件的完整删除列表 + 分区 JSON 被复制进它**每一个字节切片**的 `TFileRangeDesc`；一个被 M 个数据文件共享的删除文件，在计划里出现 **M×k 次**。大 MOR 扫描的计划体积因此多出 MB 级——FE 序列化付一次、BE 反序列化再付一次。CPU 侧还有 `delete.toThrift()`（`:415`；实现在 `IcebergScanRange.java:681`）每范围重转一遍（"第二次转换"）。 + +--- + +## 解决方案的调用栈与问题 + +方向：**扫描节点级的删除文件字典 + 每范围索引**（去重）。 + +``` +FE 发送端： + createScanRangeLocations() PluginDrivenScanNode.java:1724 + └─ scanProvider.populateScanLevelParams(params, props) :1732 ← 已存在的"扫描节点级"通用挂钩 + └─ (iceberg 新 override) 把整次扫描的去重删除文件表 + 写进 params.iceberg_delete_dict : list ← 每个删除文件只序列化 1 次 + setScanParams(rangeDesc, split) → populateRangeParams(...) [每范围] + └─ fileDesc.setDeleteFileIndices([3, 7, 12]) ← 每范围只带几个 int 索引，不带完整列表 + +BE 读取端： + iceberg_reader_mixin.h + dict = get_params().iceberg_delete_dict ← 扫描级读一次 + for (i : table_desc.delete_file_indices): ← 索引 → 字典解析 + 分桶(dict[i]) +``` + +关键先例（证明形状可行）：`TFileScanRangeParams` 里 paimon 的字段 27/30 就是"扫描节点级、避免每分片重复序列化"（`PlanNodes.thrift:549-556` 注释原话 *"Set at ScanNode level to avoid redundant serialization in each split"*），FE 侧正是通过 `ConnectorScanPlanProvider.populateScanLevelParams`（`:474`；`PaimonScanPlanProvider.java:1355` 已用同法）挂上去的。 + +**方案自身的问题（正是它需要用户签字的原因）**： + +1. **这是协议演进，不是干净的回归修复**。要动**三层**：FE 发送端 + thrift schema（新字段）+ **BE 读取端**（`iceberg_reader_mixin.h` 改成"先按索引查字典再分桶"）。触碰"改 BE + 改协议"铁律 D → **必须用户先签字**。 + +2. **跨版本兼容矩阵是最难、也是承载性的一环**。集群里 FE/BE 可能版本不一： + - **老 BE 配新 FE**：老 BE 不认新字典字段，只读 `delete_files`——新 FE 要么继续内联发一份（那就没省字节）、要么按"BE 能力位"判断对方支持才发字典； + - **新 BE 配老 FE**：老 FE 只发 `delete_files`，新 BE 要能回退到内联读。 + - 这套"双发/能力位门控 + 双向回退"是真正的工作量，不能跳过。 + +3. **验证最重**：FE 单测（字典+索引与内联语义等价）+ BE reader 单测（索引解析 + 内联回退）+ 真实 MOR 表 e2e（position/equality/deletion vector 三类删除结果不变）+ **混版本兼容矩阵**（老BE↔新FE、新BE↔老FE 都要正确读到删除行）。 + +4. **一个澄清**：FE 发送端其实**不需要动 fe-core**（`populateScanLevelParams` 这个连接器无关挂钩已存在、paimon 已用同法），所以这块不碰"fe-core 加面"。难点全在 **BE + 兼容**。 + +5. **有个 FE-only「半赢」但基本没用**：因为上一步已让同文件的 k 个切片在 FE 堆里共享同一份删除列表，可顺手把 `toThrift()` 也缓存复用（k→1 次转换、省点 FE CPU/堆）。但 thrift 序列化每遇一次引用**仍会把整个结构完整写一遍**，所以**线路字节一个都不少**——只省 FE 的 CPU/堆，还要给不可变可序列化类加可变缓存，多半不值得单做。 + +--- + +## 示例 + +设一个 MOR 表：一个 equality delete 文件 `del-a.parquet`（路径 + 边界 + field_ids 序列化后约 200 字节），被 1000 个数据文件共享；每个数据文件平均切成 4 个字节切片。 +- **当前**：这一个删除文件被序列化 1000 × 4 = **4000 次** ≈ 800 KB，全是重复的同一份；表里若有几十个这样的共享删除文件 → **几 MB 的重复**塞进发给每台 BE 的计划，FE 建 + BE 解析双向付费。 +- **字典方案后**：`del-a.parquet` 在扫描级字典里**只序列化 1 次**（200 字节）；4000 个范围各带一个 4 字节索引 ≈ 16 KB。这一个删除文件从 800 KB 降到 ~16 KB。 +- **代价对照**：换来的是要改 thrift + BE reader + 扛住"新旧 FE/BE 混跑都要正确读到删除行"的兼容矩阵——删错或读漏删除文件 = 查询结果错，所以兼容那一环必须做扎实。 + +--- + +## 立项前须确认 + +- [ ] 用户签字：**同意做协议演进**（改 thrift + 改 BE 读取端）。 +- [ ] 定兼容策略：双发 vs BE 能力位门控；确认老BE↔新FE、新BE↔老FE 双向都能正确读删除。 +- [ ] 定新字段布局：字典挂 `TFileScanRangeParams`（仿 paimon 27/30）+ 每范围索引挂 `TIcebergFileDesc`。 +- [ ] 全套验证：FE 等价单测 + BE reader/回退单测 + 真 MOR e2e（三类删除）+ 混版本矩阵。 + +## 关联 + +审计原始证据 / 逐项调用链：`plan-doc/reviews/perf-audit-fe-connector-iceberg-2026-07-17*`（该簇 findings，自述"协议演进非回归修复"）；任务空间 `plan-doc/perf-hotpath-iceberg/`。 diff --git a/plan-doc/per-statement-table-owner-port/HANDOFF.md b/plan-doc/per-statement-table-owner-port/HANDOFF.md new file mode 100644 index 00000000000000..f3a411088666a7 --- /dev/null +++ b/plan-doc/per-statement-table-owner-port/HANDOFF.md @@ -0,0 +1,36 @@ +# 🤝 Session Handoff —— 每语句表加载归属者 · 移植到其它连接器 + +> **滚动文档**：每 session 结束覆盖式更新，只留下一个 session 必须的上下文。 +> 开场必读顺序、模板、铁律见 [`README.md`](./README.md)；进度见 [`tasklist.md`](./tasklist.md)。 + +--- + +# 🆕 本空间刚建立（2026-07-19），下一个 session = **起步：确认范围 + 逐候选 recon** + +## 第一件事（**不是动码**） +1. 读 `README.md`（尤其"已就位地基"+"连接器侧移植模板 5 步"+"候选连接器表"）。 +2. 读 iceberg 蓝本：`../perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-summary.md` + 代码 `fe-connector-iceberg/.../IcebergStatementScope.java`（唯一现成范本）。 +3. **向用户复述范围并确认**：候选 = paimon(高) / hive-hms(中-高) / hudi(中)；maxcompute/es/jdbc/trino 大概率排除（0 metastore loadTable fan-out）。先定"做哪几个、从哪个起"。 +4. 选定第一个连接器后，按 README「单连接器立项流程」step 1 = **recon**（数一条 DML 加载同表几次 / 现有缓存边界 / 有无胖句柄+跨臂暂存），产出现状图，再写设计交用户确认，再动码。 + +## 关键已知（省得重新发现） +- **地基已就位、勿再改**：`ConnectorStatementScope` + `ConnectorSession.getStatementScope()`（fe-connector-api）+ `ConnectorStatementScopeImpl` + `StatementContext` 懒建/重置 + `ConnectorSessionImpl` 构造期捕获 + `ExecuteCommand` 重置（fe-core）。**移植=纯连接器侧、免 fe-core 两段验**。 +- **目前只有 iceberg 用了该 SPI**（`grep -rln getStatementScope fe/fe-connector/*/src/main/java` 只出 iceberg + api）。 +- **初摸数据**（2026-07-19，`loadTable`/`getTable` 触点文件数）：paimon 4 / hive 3 / hudi 1 / maxcompute·es·jdbc·trino 0。写路径连接器：hive、iceberg、jdbc、maxcompute、hudi 有 `getWritePlanProvider`/写面；**paimon 未在连接器层 override `getWritePlanProvider`**（写路径结构待确认）。 + +## 起步须与用户确认的点 +- **范围**：只做 catalog-backed 三家（paimon/hive/hudi），还是也看读-only 连接器？（建议：先只做多载明显的，读-only 复核后排除）。 +- **顺序**：建议从 **paimon** 起（多载最像 iceberg、`fe-connector-cache` 框架副本已在），或从写路径最清晰的 **hive** 起。 +- **hive 网关特殊性**：hms 网关按 handle 选 sibling provider（委派 iceberg/hudi），且 hms 休眠（未进 `SPI_READY_TYPES`）——设计时要专门处理"网关自身 vs 委派 sibling"的作用域归属。 + +## 铁律提醒 +- 地基勿再改；移植只写连接器侧；暴露地基缺口先停手交 review。 +- 作用域跨用户即泄漏——凡有 session=user/凭证语义的连接器，复核共享安全（iceberg 判据：授权在 load 调用里，缓存命中绕过它）。 +- surgical：模板里"拆胖句柄""下沉暂存"仅当该连接器真有对应物才做。 + +--- + +# 🗂 遗留 / 关联 +- iceberg 蓝本任务空间：`../perf-hotpath-iceberg/`（PERF-07 权威设计/小结）。 +- 架构记忆：`iceberg-table-resolution-cache-scoping`。 +- e2e 一律留各连接器进 `SPI_READY_TYPES` 切换阶段统一补（对齐 `hms-iceberg-delegation-needs-e2e`）。 diff --git a/plan-doc/per-statement-table-owner-port/README.md b/plan-doc/per-statement-table-owner-port/README.md new file mode 100644 index 00000000000000..41c6fdaab3dbf9 --- /dev/null +++ b/plan-doc/per-statement-table-owner-port/README.md @@ -0,0 +1,88 @@ +# 📦 任务空间 —— 把「每语句表加载归属者」范式从 iceberg 移植到其它连接器 + +> **独立任务空间**。目标：把 iceberg 上已落地的「一条语句里同一张表只加载一次、读/扫描/写共享」范式（PERF-07 蓝本），移植到其它 catalog-backed 连接器。 +> **重要前提**：这套范式依赖的**中性 fe-core / fe-connector-api 地基已经就位**（PERF-07 建）——移植是**纯连接器侧工作，无需再改 fe-core**。 +> 开场必读顺序、单连接器立项流程、约束铁律见下。协作规范沿用 [`../AGENT-PLAYBOOK.md`](../AGENT-PLAYBOOK.md)。 + +--- + +## 🚩 一句话背景 + +新框架下，一条 DML（尤其 DELETE/MERGE）里同一张表会被**反复远端加载**：读元数据、扫描规划、写成形、开事务各自 `loadTable`，最糟一条语句 3~5 次。iceberg 已通过 PERF-07 修掉（读/扫描/写/beginWrite 共享同一「每语句表加载归属者」，整条语句一张表只加载一次）。本空间把同一范式推广到其它同样"metastore/catalog 反复加载同表"的连接器。 + +> ⚠️ 诚实定位（对齐 iceberg 蓝本）：这类改动**性能收益有限**（写侧 ≈0，读侧靠既有缓存已覆盖一部分），**主价值 = 架构连贯 + 删重复加载/单例/胖句柄**。是否对某连接器值得做，取决于它真实的多载程度——**逐连接器复核后再定，不是照单全上**。 + +--- + +## 🧱 已就位的可复用地基（**勿再改，直接复用**） + +PERF-07 已在 fe-core / fe-connector-api 落地一套**连接器无关**的每语句作用域基建（commit `97bdcd6bdbe`）： + +| 组件 | 位置 | 作用 | +|---|---|---| +| `ConnectorStatementScope`（SPI + `NONE`） | `fe-connector-api/.../connector/api/ConnectorStatementScope.java` | 中性接口：` T computeIfAbsent(String key, Supplier)`；`NONE`=不缓存(逐次加载)，离线/无上下文默认 | +| `ConnectorSession.getStatementScope()` | `fe-connector-api/.../connector/api/ConnectorSession.java` | 默认方法→`NONE`；连接器经它够到每语句作用域 | +| `ConnectorStatementScopeImpl` | `fe-core`（CHM 背书） | 生产实现，挂 `StatementContext` | +| `StatementContext` 懒建字段 + `resetConnectorStatementScope()` | `fe-core` | 唯一贯穿整条语句的对象；不在 close/release 清、随 GC | +| `ConnectorSessionImpl`/`Builder` **构造期捕获** | `fe-core` | 因扫描流式/分批 off-thread 复用同一 session、实时读 thread-local 会失效，故构造期捕获作用域引用 | +| `ExecuteCommand` 一行重置 | `fe-core` | 预编译 EXECUTE 复用同一 `StatementContext`，每执行重置作用域 | + +**移植一个连接器 = 只写连接器侧代码**（不再碰 fe-core，即不再触"fe-core 只减不增"铁律 A）。 + +--- + +## 🧩 连接器侧移植模板（以 `IcebergStatementScope` 为范本） + +对每个目标连接器，镜像 iceberg 的做法（`fe-connector-iceberg/.../IcebergStatementScope.java` 是唯一现成范本）： + +1. **加连接器私有 `XStatementScope` helper**：`sharedTable(session, db, tbl, loader)`，键形如 `x.table:catalogId:db:tbl:queryId`；`session==null`（NONE）时逐次加载（等价无缓存）。值存 RAW 表对象（fe-core 不认连接器类型，保持 connector-agnostic）。 +2. **四处表解析全走 `sharedTable`**：读元数据 / 扫描规划 / 写成形 / `beginWrite`——同一语句同一表命中同一次加载。每处 loader 各保留原授权（`newXBackedOps(session)`，一语句=一用户=一凭证）。 +3. **（若有）拆胖句柄**：若该连接器像 iceberg 那样在 handle 上挂了 `resolvedTable` 之类的"同 handle 内 memo"，评估是否随之下沉到作用域、让句柄回归纯坐标。**没有就不做**（surgical）。 +4. **（若有）把每语句暂存下沉到作用域**：iceberg 把删除清单暂存（`IcebergRewritableDeleteStash`）从单例下沉到作用域同键 map 并整删该类。目标连接器若有类似"扫描填、写读"的跨臂暂存，同样下沉；没有就跳过。 +5. **响亮失败守卫**：若存在"作用域缺失=静默错误"的路径（iceberg 的 v3 行级 DML + NONE），加 fail-loud（NONE 下抛，消息含 "per-statement scope"）。生产恒有 `StatementContext`，仅离线触发。 + +**测试守门**（镜像 iceberg）： +- 每语句加载计数守门：读+扫描（+写）共享作用域 → 远端 `loadTable` 计数 **1**；对照 NONE → N。 +- 作用域隔离：同键读写同实例 / 跨 queryId 隔离（预编译重执行）/ 跨 catalogId 隔离（跨-catalog 语句）/ NONE 逐次。 +- e2e：留连接器进入 `SPI_READY_TYPES` 的切换阶段统一补（对齐 iceberg 的 e2e 欠账惯例）。 + +--- + +## 🎯 候选连接器（2026-07-19 初摸，**范围待下 session 复核确认**） + +| 连接器 | 写路径 | `loadTable`/`getTable` 触点 | 候选度 | 备注（待 recon 核实） | +|---|---|---|---|---| +| **paimon** | 有（MTMV/写；`getWritePlanProvider` 未在连接器层 override，写路径结构待确认） | **4 文件**（最多） | **高** | 读侧多载最像 iceberg；`fe-connector-cache` 已复制框架副本 | +| **hive / hms** | 有（`HiveWritePlanProvider`） | 3 文件 | **中-高** | plain-hive 读写活；hms 网关委派 iceberg/hudi 兄弟（休眠，未进 `SPI_READY_TYPES`）；**网关按 handle 选 sibling 的特殊性要单独设计** | +| **hudi** | 有（`HudiConnectorMetadata` 写面） | 1 文件 | **中** | 读为主（MTMV 新鲜度已提升）；写臂/多载程度待确认 | +| maxcompute / es / jdbc / trino | 部分有写 | **0**（不走 metastore `loadTable` fan-out） | **低 / 大概率不适用** | 解析模型不同（表对象连接器侧缓存 / 无每语句重载）；下 session 确认后**排除**居多 | + +> **第一步不是动码，是逐连接器 recon**：确认每个候选真实的"一条语句加载同表几次"、现有缓存覆盖哪段、是否有胖句柄/跨臂暂存。复核可能把某连接器**判为不必做**（如 iceberg PERF-05/10 就被复核缩小/暂缓过）。 + +--- + +## 🔁 单连接器立项流程（一次一个连接器，对齐 `step-by-step-fix`） + +1. **recon（动码前）**：grep 该连接器读/扫描/写/beginWrite 的表解析调用链，数"一条 DML 加载同表几次"、现有缓存边界、有无胖句柄/跨臂暂存。产出该连接器的现状图。 +2. **写设计** `designs/PORT--design.md`：Problem / 现状调用链 / 移植方案（按上面 5 步模板裁剪）/ 与 iceberg 蓝本差异 / Risk / Test Plan（含加载计数守门）。 +3. **设计红队**（对抗 review，clean-room 偏好）：至少一个独立视角挑刺——重点核**授权/凭证隔离**（作用域跨用户是否泄漏，参照 iceberg 的 session=user/vended 判定）与**快照/OCC 一致性**。 +4. **实现**：纯连接器侧、最小改动、镜像 `IcebergStatementScope`。 +5. **验证**：连接器 UT 全绿 + 加载计数守门；纯连接器改动**免 fe-core 两段验**（地基已在）。 +6. **独立 commit** + **写小结** `designs/PORT--summary.md` + 勾 `tasklist.md` + 追加 `progress.md` + 覆盖 `HANDOFF.md`。 + +--- + +## 🧱 约束铁律 + +1. **地基勿再改**：`ConnectorStatementScope`/`getStatementScope()`/`ConnectorStatementScopeImpl`/`StatementContext` 已就位，移植**只写连接器侧**——不再碰 fe-core（不重新触铁律 A）。若某连接器暴露地基缺口，先停手交 review，别顺手改 fe-core。 +2. **作用域跨用户即泄漏**：作用域按 `catalogId+queryId`（+db/tbl）建键、存 RAW、随 session 生死；但**缓存命中会绕过 per-user loadTable 里的授权**——凡有 `session=user`/凭证语义的连接器，复核作用域共享是否安全（参照 iceberg 判据：授权发生在 load 调用里，缓存命中绕过它=元数据泄漏）。 +3. **连接器不解析属性 / 通用节点 connector-agnostic**：沿用本项目一贯铁律；作用域值为 `Object`，fe-core 不认连接器类型。 +4. **surgical**：模板 5 步里"拆胖句柄""下沉暂存"仅当该连接器真有对应物才做；没有就不做。 + +--- + +## 🔗 与 iceberg 蓝本的关系 + +- **权威范本**：`plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-{design,summary}.md`；架构结论落 memory `iceberg-table-resolution-cache-scoping`。 +- **代码范本**：`fe-connector-iceberg` 的 `IcebergStatementScope` + 四处 `resolveTable*` 走它 + fail-loud（commit `ea7fd1f6e7a`）。 +- 本空间是那套范式的**推广执行区**，与 `perf-hotpath-iceberg` 任务空间平行、不混流。 diff --git a/plan-doc/per-statement-table-owner-port/progress.md b/plan-doc/per-statement-table-owner-port/progress.md new file mode 100644 index 00000000000000..58bdddb3ad8400 --- /dev/null +++ b/plan-doc/per-statement-table-owner-port/progress.md @@ -0,0 +1,13 @@ +# Progress Log —— 每语句表加载归属者 · 移植到其它连接器 + +> **Append-only**：只追加、不覆盖（覆盖式状态在 HANDOFF/tasklist）。 + +--- + +## 2026-07-19 — session 0：建任务空间 + +- 用户拍板：把 iceberg 的 PERF-07「每语句表加载归属者」整体性改动移植到其它连接器；**本 session 只建跟踪文档空间，实际处理留下一个 session**。 +- 读准 iceberg 蓝本（PERF-07 summary）：可复用地基 = 中性 `ConnectorStatementScope`(fe-connector-api) + `ConnectorSession.getStatementScope()` + `ConnectorStatementScopeImpl`/`StatementContext`/`ConnectorSessionImpl` 构造期捕获/`ExecuteCommand` 重置(fe-core)，**已随 PERF-07 落地**（commit `97bdcd6bdbe`）。→ 关键结论：**移植是纯连接器侧工作，无需再改 fe-core**（不再触铁律 A）。连接器侧范式 = `IcebergStatementScope` helper + 四处 `resolveTable*` 共享 + (可选)拆胖句柄 + (可选)下沉跨臂暂存 + (可选)fail-loud。 +- 初摸候选（grep）：只有 iceberg 用了该 SPI；`loadTable`/`getTable` 触点 paimon 4 / hive 3 / hudi 1 / maxcompute·es·jdbc·trino 0。→ 候选 = paimon(高)/hive-hms(中-高)/hudi(中)；读-only 四家大概率排除，待复核。 +- 落地文件：`README.md`（用途 + 已就位地基 + 连接器侧 5 步模板 + 候选表 + 单连接器立项流程 + 铁律）、`tasklist.md`（PORT-01~04 全待 recon）、`HANDOFF.md`（下一步 = 确认范围 + 逐候选 recon）、`progress.md`（本文件）、`designs/`（空）。 +- **未动任何产品代码**。**下一步**：见 HANDOFF —— 下个 session 起步先与用户确认范围/顺序，再对第一个连接器 recon。 diff --git a/plan-doc/per-statement-table-owner-port/tasklist.md b/plan-doc/per-statement-table-owner-port/tasklist.md new file mode 100644 index 00000000000000..a6e0435e57f4e6 --- /dev/null +++ b/plan-doc/per-statement-table-owner-port/tasklist.md @@ -0,0 +1,25 @@ +# Task List —— 每语句表加载归属者 · 移植到其它连接器 + +> **唯一进度清单**。每完成一项随 commit 勾 `[x]` + 填状态/commit。ID 一旦分配永不复用。 +> 立项流程、约束铁律、模板见 [`README.md`](./README.md)。地基（fe-core/api SPI）已就位，移植=纯连接器侧。 +> 状态图例：⏳ 待启动 · 🔍 recon 中 · 🚧 进行中 · ✅ 完成 · 🔬 复核判不必做 · ❌ 阻塞 + +## 总览 + +- **蓝本 = iceberg（PERF-07 已完成）**：`ConnectorStatementScope`(fe-core/api，中性) + `IcebergStatementScope`(连接器) + 四处 `resolveTable*` 共享 + 拆胖句柄 + 删除暂存下沉 + v3 fail-loud。commits `97bdcd6bdbe`(fe-core) + `ea7fd1f6e7a`(iceberg)。 +- **本清单只跟踪"其它连接器"的移植**；每项**第一步是 recon**（可能复核判"不必做"）。 + +| ID | 连接器 | 候选度 | 状态 | 备注 | commit | +|---|---|---|---|---|---| +| PORT-01 | paimon | 高 | ⏳ 待 recon | loadTable 触点最多(4)；写路径结构待确认 | | +| PORT-02 | hive / hms | 中-高 | ⏳ 待 recon | plain-hive 读写活；hms 网关委派 sibling(休眠)——网关按 handle 选 provider 的特殊性要单独设计 | | +| PORT-03 | hudi | 中 | ⏳ 待 recon | 读为主(MTMV)；写臂/多载程度待确认 | | +| PORT-04 | maxcompute / es / jdbc / trino | 低 | ⏳ 待判定 | 0 个 metastore loadTable fan-out；大概率**排除**，下 session 确认后标 🔬 | | + +--- + +## 说明 + +- iceberg 本身**不在本清单**（已由 PERF-07 完成，见 `plan-doc/perf-hotpath-iceberg/`）。 +- 各连接器完成后建各自 `designs/PORT--{design,summary}.md`（不预建）。 +- **不立项**的连接器（复核判不必做）标 🔬 + 一句原因，保留占位。 From 7d7e671f84155e0c7303e92dca08830cb03508eb Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 10:19:42 +0800 Subject: [PATCH 28/47] [doc](catalog) per-statement-table-owner port: cross-connector recon conclusions + Trino-refactor groundwork MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Recon (recon + adversarial cross-check, double-signed) of every catalog-backed connector for porting iceberg's per-statement table-load-owner pattern. Verdict: no connector is worth porting now. iceberg is unique in being the only connector with a migrated row-level write path (DELETE/MERGE) — the only capability that triggers the multi-arm reload storm the pattern targets. Every other connector is read-only (paimon/hudi/es/trino) or INSERT/append-only (hive/hms, jdbc, maxcompute), and existing cross-query caches already collapse loads to ~1. Also recorded the architecture-unification analysis: the uniform SPI standard already exists (neutral getStatementScope() + ConnectorStatementScope); Trino's per-transaction-metadata model is not directly portable (Doris connectors are shared singletons; the only per-statement span is StatementContext). Altitude tiers L0-L3 laid out; paimon's future row-level-write migration is the correct trigger to extract a shared helper (L1) into fe-connector-api (no iron-rule-A breach). Per user decision: this round only records conclusions to a standalone doc; the Trino-architecture refactor (L2/L3) is deferred to a dedicated next-session discussion (groundwork staged in the doc's section 7). No product code changed. - designs/recon-findings-and-trino-refactor-groundwork.md (new, full findings) - tasklist.md: PORT-01..04 all marked recon-judged-unnecessary (paimon = future candidate) - progress.md / HANDOFF.md: session 1 recon + next-session Trino-refactor discussion Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../per-statement-table-owner-port/HANDOFF.md | 49 +++-- ...-findings-and-trino-refactor-groundwork.md | 169 ++++++++++++++++++ .../progress.md | 15 ++ .../tasklist.md | 15 +- 4 files changed, 226 insertions(+), 22 deletions(-) create mode 100644 plan-doc/per-statement-table-owner-port/designs/recon-findings-and-trino-refactor-groundwork.md diff --git a/plan-doc/per-statement-table-owner-port/HANDOFF.md b/plan-doc/per-statement-table-owner-port/HANDOFF.md index f3a411088666a7..fba41a006c7d38 100644 --- a/plan-doc/per-statement-table-owner-port/HANDOFF.md +++ b/plan-doc/per-statement-table-owner-port/HANDOFF.md @@ -1,36 +1,49 @@ # 🤝 Session Handoff —— 每语句表加载归属者 · 移植到其它连接器 > **滚动文档**：每 session 结束覆盖式更新，只留下一个 session 必须的上下文。 -> 开场必读顺序、模板、铁律见 [`README.md`](./README.md)；进度见 [`tasklist.md`](./tasklist.md)。 +> 开场必读顺序、模板、铁律见 [`README.md`](./README.md)；进度见 [`progress.md`](./progress.md)；状态见 [`tasklist.md`](./tasklist.md)。 --- -# 🆕 本空间刚建立（2026-07-19），下一个 session = **起步：确认范围 + 逐候选 recon** +# 🆕 本轮（2026-07-19 session 1）已完成：**全部连接器 recon + 架构统一性调研 → 结论全部 🔬** -## 第一件事（**不是动码**） -1. 读 `README.md`（尤其"已就位地基"+"连接器侧移植模板 5 步"+"候选连接器表"）。 -2. 读 iceberg 蓝本：`../perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-summary.md` + 代码 `fe-connector-iceberg/.../IcebergStatementScope.java`（唯一现成范本）。 -3. **向用户复述范围并确认**：候选 = paimon(高) / hive-hms(中-高) / hudi(中)；maxcompute/es/jdbc/trino 大概率排除（0 metastore loadTable fan-out）。先定"做哪几个、从哪个起"。 -4. 选定第一个连接器后，按 README「单连接器立项流程」step 1 = **recon**（数一条 DML 加载同表几次 / 现有缓存边界 / 有无胖句柄+跨臂暂存），产出现状图，再写设计交用户确认，再动码。 +## 一句话结果 +- **逐连接器复核（双签）：没有一个连接器现在值得移植**——iceberg 独特在它是**唯一迁移了行级写（DELETE/MERGE）**的连接器；别的连接器只读 / 仅追加写，没有多臂重复加载风暴，现有跨查询缓存又已把加载压到≈1。四项全部标 🔬。 +- **"统一接口标准"其实已存在**（中性 `getStatementScope()` + `ConnectorStatementScope`，新连接器天生继承）；**推荐高度 L0**（写下约定 + 登记触发点，生产逻辑零改动）。 +- **用户拍板**：本轮先把结论记成**单独文档**（已落 `designs/recon-findings-and-trino-refactor-groundwork.md`）；**重构成 Trino 架构（L2/L3）留到下个 session 专题讨论**。 +- **未动任何产品代码**（纯 plan-doc）。 -## 关键已知（省得重新发现） -- **地基已就位、勿再改**：`ConnectorStatementScope` + `ConnectorSession.getStatementScope()`（fe-connector-api）+ `ConnectorStatementScopeImpl` + `StatementContext` 懒建/重置 + `ConnectorSessionImpl` 构造期捕获 + `ExecuteCommand` 重置（fe-core）。**移植=纯连接器侧、免 fe-core 两段验**。 -- **目前只有 iceberg 用了该 SPI**（`grep -rln getStatementScope fe/fe-connector/*/src/main/java` 只出 iceberg + api）。 -- **初摸数据**（2026-07-19，`loadTable`/`getTable` 触点文件数）：paimon 4 / hive 3 / hudi 1 / maxcompute·es·jdbc·trino 0。写路径连接器：hive、iceberg、jdbc、maxcompute、hudi 有 `getWritePlanProvider`/写面；**paimon 未在连接器层 override `getWritePlanProvider`**（写路径结构待确认）。 +--- + +# ➡️ 下一个 session = **专题讨论"重构成 Trino 架构"（L2/L3）** -## 起步须与用户确认的点 -- **范围**：只做 catalog-backed 三家（paimon/hive/hudi），还是也看读-only 连接器？（建议：先只做多载明显的，读-only 复核后排除）。 -- **顺序**：建议从 **paimon** 起（多载最像 iceberg、`fe-connector-cache` 框架副本已在），或从写路径最清晰的 **hive** 起。 -- **hive 网关特殊性**：hms 网关按 handle 选 sibling provider（委派 iceberg/hudi），且 hms 休眠（未进 `SPI_READY_TYPES`）——设计时要专门处理"网关自身 vs 委派 sibling"的作用域归属。 +## 第一件事（先读，别急着给方案） +1. 读 **`designs/recon-findings-and-trino-refactor-groundwork.md`**（本轮全部结论：逐连接器复核 §2、iceberg 为何独特 §3、统一性/Trino 参照/高度分级 §4、paimon 触发点 §5、共享 helper 落点 §6、**Trino 重构预备材料 §7**）。 +2. 读架构记忆 `iceberg-table-resolution-cache-scoping`（缓存作用域纪律 + "全高度留远期"）。 + +## 讨论要点（种子，详见结论文档 §7） +- **Trino 到底要引入什么**：每语句/每事务 `ConnectorMetadata` 实例（现状是每 catalog 共享单例）+ 一个 span 生命周期管理器（Doris 现成 span 宿主=`StatementContext`）+ planner 改成经 span 取 metadata + 逐连接器把缓存迁到 per-statement 实例。 +- **必须先摆平的硬冲突**： + - **铁律 A**（删旧代码期 fe-core 只减不增）——L2/L3 几乎全是 fe-core 净增 + planner 改造 → 是否等删旧代码期结束？是否需用户单独签字？ + - 读热路径刚稳定（PERF-01~06/11），per-statement metadata 会重排读热缝、回归面大。 + - **正面收益**：per-statement 实例天然按语句/用户隔离 → 顺带解决"跨查询缓存对 session=user/vended 关闭"的历史包袱（这是 L3 相对现状的真正架构收益）。 +- **建议讨论产出**：一份"Doris 每语句/每事务 metadata 重构"的**可行性 + 分期**设计（先 span 宿主 → 逐连接器迁移 → 退役共享单例，避免一次性大爆炸）+ 明确触发条件。 + +## 关键已知（省得重新发现） +- **地基已就位、勿再改**：`ConnectorStatementScope` + `getStatementScope()`(fe-connector-api) + `ConnectorStatementScopeImpl` + `StatementContext` 懒建/重置 + `ConnectorSessionImpl` 构造期捕获 + `ExecuteCommand` 重置(fe-core)。 +- **目前只有 iceberg 用了该 SPI**（`IcebergStatementScope` + `IcebergWritePlanProvider`）。 +- **paimon = 唯一真实的将来 L1 候选**：写未迁移（老 JNI-writer INSERT 写栈在删旧代码期被删、未搬进新连接器）；已有胖句柄 `PaimonTableHandle.paimonTable` + 4 加载 seam；**加行级 UPDATE/DELETE 时才值得移植**，届时抽共享 helper（落 `fe-connector-api`，签名/6 处迁移见结论文档 §6）。 ## 铁律提醒 -- 地基勿再改；移植只写连接器侧；暴露地基缺口先停手交 review。 -- 作用域跨用户即泄漏——凡有 session=user/凭证语义的连接器，复核共享安全（iceberg 判据：授权在 load 调用里，缓存命中绕过它）。 -- surgical：模板里"拆胖句柄""下沉暂存"仅当该连接器真有对应物才做。 +- 地基勿再改；移植（若做）只写连接器侧；暴露地基缺口先停手交 review。 +- L2/L3 碰铁律 A（fe-core 净增）——**属独立立项 + 需用户签字**，不在本"纯连接器侧移植"任务的默认范围内。 +- 作用域跨用户即泄漏——凡有 session=user/凭证语义的连接器，复核共享安全（本轮已确认现有各连接器均目录级单一身份、不泄漏）。 --- # 🗂 遗留 / 关联 +- 本轮全部结论：`designs/recon-findings-and-trino-refactor-groundwork.md`。 - iceberg 蓝本任务空间：`../perf-hotpath-iceberg/`（PERF-07 权威设计/小结）。 - 架构记忆：`iceberg-table-resolution-cache-scoping`。 +- 本轮调研原始返回：workflow journal `wf_e89cf92e-ff3`（逐连接器）+ `wf_4802a3d2-1c9`（统一性三专项），路径见结论文档 §8。 - e2e 一律留各连接器进 `SPI_READY_TYPES` 切换阶段统一补（对齐 `hms-iceberg-delegation-needs-e2e`）。 diff --git a/plan-doc/per-statement-table-owner-port/designs/recon-findings-and-trino-refactor-groundwork.md b/plan-doc/per-statement-table-owner-port/designs/recon-findings-and-trino-refactor-groundwork.md new file mode 100644 index 00000000000000..e7a6327fd1dc5f --- /dev/null +++ b/plan-doc/per-statement-table-owner-port/designs/recon-findings-and-trino-refactor-groundwork.md @@ -0,0 +1,169 @@ +# 每语句表加载归属者 · 跨连接器复核结论 + Trino 重构议题预备 + +> **本文件定位**：把"把 iceberg 的每语句表加载归属者移植到其它连接器"这件事的**全部调研结论**固化到一处，供后续 session 直接接手。 +> **本轮不动任何生产代码**。用户拍板：先把结论记成文档；**重构成 Trino 架构的问题留到下一个 session 专题讨论**（见 §7）。 +> 蓝本/地基/铁律见同目录 [`README.md`](../README.md)；进度见 [`progress.md`](../progress.md) / 状态见 [`tasklist.md`](../tasklist.md)。 + +--- + +## 0. 一句话结论 + +- **逐连接器复核（recon + 独立对抗复核，双签）：没有一个连接器现在值得移植。** iceberg 之所以值得，唯一原因是它是本仓库里**唯一迁移了行级写（DELETE/MERGE）**的连接器——行级写让同一张表在一条语句里被"读+扫描+写成形+开事务"反复远端加载 3~5 次，这才是那套改造的靶子。别的连接器要么在 Doris 侧只读、要么仅 INSERT 追加写，没有这个重复加载风暴；现有的跨查询缓存又已把加载压到 ≈1 次。 +- **"统一接口标准"其实已经存在**：中性 SPI `ConnectorSession.getStatementScope()` + `ConnectorStatementScope`(带 `NONE` 默认)。每个连接器天生继承，离线自动退化成安全的逐次加载。新连接器**不是没有标准可循**。 +- **真正的将来触发点 = paimon 加行级写**。paimon 现在已经是"iceberg 改造前"的形状（有胖句柄 + 多加载 seam），只差写臂；一旦加 UPDATE/DELETE 就会复现风暴，届时移植才有价值，且那是抽取共享 helper 的正确时机（有 iceberg + paimon 两个真实用户）。 +- **推荐高度 = L0**（写下约定 + 登记触发点，生产逻辑零改动）；**L1**（抽共享 helper）留到 paimon 加写时；**L2/L3（Trino 式重构）** 是远期、跨切、与删旧代码期铁律 A 冲突的独立项目——**下个 session 专门讨论**。 + +--- + +## 1. 背景（给未来读者的最小上下文） + +新 SPI 框架下，一条 DML（尤其 DELETE/MERGE）里同一张表被反复远端 `loadTable`：读元数据、扫描规划、写成形、开事务各自加载，最糟一条语句 3~5 次。连接器 session 一条语句内被重建约 26 次，缓存挂它即死。唯一贯穿整条语句的对象是 fe-core 的 `StatementContext`。 + +iceberg 已通过 **PERF-07** 修掉（commit `97bdcd6bdbe` 建地基 + `ea7fd1f6e7a` 连接器侧）： +- **中性地基（已就位、勿再改）**：`ConnectorStatementScope`(fe-connector-api) + `ConnectorSession.getStatementScope()` + `ConnectorStatementScopeImpl`/`StatementContext` 懒建/重置 + `ConnectorSessionImpl` 构造期捕获 + `ExecuteCommand` 重置(fe-core)。 +- **连接器侧范式**：`IcebergStatementScope` helper（`sharedTable` 键 `iceberg.table:catalogId:db:tbl:queryId` + `rewritableDeleteSupply`）+ 四处表解析共享 + 拆胖句柄（删 `IcebergTableHandle.resolvedTable`）+ 下沉跨臂删除暂存（删 `IcebergRewritableDeleteStash` 单例）+ v3 DML under NONE fail-loud。 + +**移植一个连接器 = 纯连接器侧工作，不再碰 fe-core**（不触铁律 A）。 + +--- + +## 2. 逐连接器复核结论（recon + 对抗复核，双签，2026-07-19） + +> 方法：每个候选一个深度 recon agent（数一条 DML 真实加载几次 / 现有缓存边界 / 胖句柄 / 跨臂暂存 / 鉴权凭证语义），再配一个**独立对抗复核 agent**（专挑"加载被现有缓存兜住""共享泄漏"两类刺）。以下均为双签、置信=高。 + +| 连接器 | 有写入面？ | 一条 DML 真实远端加载 | 胖句柄 | 跨臂暂存 | 凭证泄漏 | 结论 | +|---|---|---|---|---|---|---| +| **paimon** | ❌ 未迁移（只读） | DML=0；SELECT≈1 | ✅ 有 `PaimonTableHandle.paimonTable`（载荷型，非纯 memo） | ❌ | 无（目录级单一身份；vend 仅用于下游存储） | **将来候选**（触发点=加行级写） | +| **hive / hms 网关** | ⚠️ 仅 INSERT/OVERWRITE | DML=N/A；写≈0~1 | ❌（纯坐标+扁平标量） | ❌ | 无（连接器级单一身份） | **不必做** | +| **hudi** | ❌ 只读 | DML=0；读侧 ~4-6 次未缓存 metaClient 构建 | ❌ | ❌ | 无 | **不必做** | +| **maxcompute** | ⚠️ 仅 INSERT 追加 | 低（句柄带惰性表代理，写复用不重载） | ❌ | ❌ | 无 | **排除** | +| **es** | ❌ 只读 | 读侧 mapping 有重取（已被 fe-core schema 缓存兜） | ❌ | ❌ | 无 | **排除** | +| **jdbc** | ⚠️ 仅 INSERT 追加 | 低（纯坐标句柄，写复用坐标） | ❌ | ❌ | 无 | **排除** | +| **trino** | ❌ 只读 | 读侧 `getTableHandle` 每 seam 重解析、**最贵且未缓存** | 同句柄内 memo | ❌ | 无 | **排除**（读侧优化候选，另立议题） | + +### 关键证据（file:method，便于将来核验） + +**hive / hms 网关（不必做，置信高）** +- 无行级写：`HiveWritePlanProvider.supportedOperations()` = `EnumSet.of(INSERT, OVERWRITE)`；full-ACID 写在 `HiveConnectorTransaction.rejectTransactionalWrite`(beginWrite:210) 硬拒。DML 风暴结构性不存在。 +- 加载已被**跨查询**缓存兜住：读/写/beginWrite 三处 `getTable` 全走 `CachingHmsClient.tableCache`（跨查询、TTL 24h、按 (db,table) 键）→ 实际远端 ≈0~1 次，是"每语句作用域"的**超集**。已核实缓存**确实接线**（`HiveConnector.createClient:588` = `wrapWithCache(new ThriftHmsClient(...))`；`hms` 在 `CatalogFactory.SPI_READY_TYPES`）——代码里多处 "dormant/未进 SPI_READY_TYPES" 的 javadoc 是 cutover 后的**过期注释**（doc-hygiene 待清）。 +- 无胖句柄（`HiveTableHandle`=纯坐标+扁平标量）；无跨臂暂存（`HiveReadTransactionManager` 是 ACID 读锁生命周期管理器，非扫描→写桥）。 +- **网关委派已查清**：网关只对普通 hive 表自己加载；遇 iceberg/hudi-on-HMS 表只做 **1 次廉价格式探测加载**（`getTableHandle:343`）后委派兄弟连接器，兄弟自带作用域——**网关自己不需要任何作用域**，也不得重复包裹委派加载。 + +**paimon（将来候选，置信高）** +- **写未迁移**：`PaimonConnector` 不 override `getWritePlanProvider`（→ null → `supportedWriteOperations()` 空 → INSERT/DELETE/MERGE 全拒）；代码注释 `PaimonConnector.java:294` 明写 "paimon write is not migrated"。 +- 加载已被句柄 memo + SDK 缓存压到 ≈1：`getTableHandle:185` 单次 `catalogOps.getTable` 后 `setPaimonTable` 存到 transient 句柄；后续 ~8 处 `resolveTable`/`resolveScanTable` 走快路径。跨查询由 paimon SDK `CachingCatalog` 去重。 +- **胖句柄存在**：`PaimonTableHandle.paimonTable`(transient, PaimonTableHandle.java:89) 是 `IcebergTableHandle.resolvedTable` 的直接同款。但**是载荷型非纯 memo**：`withBranch` 故意置 null 让分支重载不同表、`withScanOptions` 拷贝它、承担查询内 Table 实例一致性——现在删它换作用域 map 是**平移（等价功能、无减载）+ 有回归面**。 +- 无跨臂暂存（无写臂）。鉴权=目录级单一身份；vended 凭证只用于下游 per-scan-range 存储读、不烘进 Table 对象，故共享**不泄漏**。 + +**hudi（不必做，置信高）** +- 只读：不 override 写入面，`beginTransaction`(HudiConnectorMetadata.java:397) 抛 "Hudi tables are read-only"；INSERT/DELETE/MERGE 在准入门被拒。 +- 5 步模板 4 步空操作。唯一真实重复成本=读路径 ~4-6 次**未缓存的 `HoodieTableMetaClient` 构建**（每 seam 从 basePath 重建）——但**形状不对**（不是"表加载归属者"）、**metaClient 可变**（`getSchemaFromMetaClient` 调 `reloadActiveTimeline()`）、**鉴权/TCCL 上下文错配**（metadata 臂在 `metaClientExecutor.execute` 的 doAs 里、planScan 在 fe-core 扫描线程内联无 doAs），且主成本（FileSystemView / MDT `getAllPartitionPaths`）压根不是表对象加载。 + +**maxcompute·es·jdbc·trino（排除，对抗复核确认非橡皮图章）** +- 均无行级 DML：es/trino 只读、jdbc/maxcompute 仅 INSERT-append 且写成形复用句柄/坐标不二次加载。无跨臂暂存、无 v3 复活风险、凭证目录级。 +- 诚实指出：**读路径确有"每 seam 重解析"未缓存**——trino 最重（`getTableHandle`+`getColumnHandles`+N×`getColumnMetadata` 每 seam 重跑）、es 次之（mapping 重取）。但这正是蓝本明确打折的"读侧、收益有限、已被 `SchemaCacheValue` 兜一部分"那类，且缺写侧/暂存/胖句柄的架构收益。**trino 标记为最弱排除**——若将来单独立"读侧去重"新任务，它是这组里唯一有实打实重解析成本的。 + +### 一个不改结论的开放问题 +- paimon/hive 的句柄是否跨 fe-core 边界被重建（使加载数从 1 涨到 2-3）尚未 trace 确认。但即便最坏情况，那些多出来的加载**也全部命中现有跨查询缓存**，不改变任何"不必做"的结论。 + +--- + +## 3. 核心洞察：iceberg 为什么独特 + +那套改造 5 步（① sharedTable ② 四处 seam ③ 拆胖句柄 ④ 下沉暂存 ⑤ fail-loud）的价值，几乎全押在 **③④⑤ 依赖"行级写"** 上： + +- **只有 iceberg 迁移了行级写（DELETE/MERGE）**，它同时读 + 写同一张表，才有"写成形×N + beginWrite"这些绕过读缓存的加载臂，以及"扫描填、写读"的跨臂删除暂存。 +- 其它连接器：paimon/hudi/es/trino **只读**、hive/hms **仅 INSERT/OVERWRITE**、jdbc/maxcompute **仅 INSERT-append**。对它们 ①② 要么冗余（现有缓存已兜）、③④⑤ 基本是空操作。 + +因此"逐连接器复核判不必做"不是保守，而是**iceberg 本就是特例**。 + +--- + +## 4. 架构统一性分析（回答用户"是否有必要改造和统一"） + +### 4.1 统一接口其实已存在 +Doris 版的统一契约 = 中性 SPI `ConnectorSession.getStatementScope()` + `ConnectorStatementScope`（`computeIfAbsent(key, loader)` + `NONE` 默认）。**每个连接器天生继承，离线安全退化。** `IcebergStatementScope` 只是 40 行的连接器私有便利包装。所以"新连接器没有统一标准"这个担心其实不成立——标准已发布，缺的只是"更强的强制/便利外壳"。 + +### 4.2 Trino 参照模型：统一靠"引擎自持的每事务生命周期"，不是一个缓存类 +- Trino 每条语句都是一个事务；引擎为每个被访问的 catalog **懒造一个"每事务 `ConnectorMetadata`"实例并 memoize 到事务上**（`IcebergTransactionManager`/`HiveTransactionManager` 内的 per-catalog `MemoizedMetadata`），commit/rollback 时销毁。 +- 连接器把加载到的表**缓存在这个实例上**（iceberg `IcebergMetadata.tableMetadataCache`、hive per-transaction metastore 的 `tableMetadataCache`）。`getTableHandle`/`getTableMetadata`/`getColumnHandles`/统计/`beginInsert`/`finishInsert` 都重入**同一个** metadata → 命中缓存 → 远端 `loadTable` 只发一次。 +- **统一的是"共享生命周期"（去重的 span=事务、活满 span 的对象=metadata 都由引擎供给和销毁），"cache-on-metadata" 是自然掉出来的约定**；新连接器实现 `ConnectorMetadata` 即免费获得。 + +### 4.3 为什么 Trino 模型不可直接移植到 Doris +Doris 缺 Trino 的每一个结构前提： +- **无每事务 `ConnectorMetadata`**：Doris 连接器/`*ConnectorMetadata` 是**长期共享单例**（每 catalog 一个、从不按语句造），挂每语句可变缓存会跨语句/跨用户泄漏（正因如此跨查询 `IcebergTableCache` 对 `session=user`/vended 关闭）。 +- `ConnectorSession` **不是稳定 span**（一条语句重建约 26 次，挂它即死）。 +- **DML 事务对象出现得晚、且只在写臂**（`*ConnectorTransaction` 在 beginWrite 才建），无法承载"读+扫描+写成形"整条 span。 +- **唯一活满整条语句、连接器够得到的对象 = fe-core `StatementContext`**，只能经中性 `getStatementScope()` 够到。所以 Doris 必须把 Trino 模型**反过来**：不是"引擎把每事务 metadata 发给连接器"，而是"连接器向上够 fe-core 的 span"。 +- 全盘照搬会撞铁律 A：给 14 个连接器引入每事务 metadata + TransactionManager 生命周期 + 改 planner 每语句取 metadata = 删旧代码期的 fe-core 大净增。 + +**结论：现有 `ConnectorStatementScope` + `getStatementScope()` + `NONE` 已是 Trino 思想在 Doris 铁律下能落地的最高、最可移植高度。** + +### 4.4 高度分级（L0–L3） +| 高度 | 内容 | 现在值不值 | +|---|---|---| +| **L0** | 只把统一**约定写成文**（键格式 `<类型>.table:catalogId:db:tbl:queryId`、NONE=逐次加载、扫描→写跨臂在 NONE 下 fail-loud）+ 登记触发点；生产逻辑零改动 | ✅ **推荐（现在做）** | +| **L1** | 把 iceberg 私有 helper 提升为 `fe-connector-api` 共享 util、迁移 iceberg 6 处调用（纯连接器侧、fe-core 不动、不碰铁律 A） | ⚠️ 留到 paimon 加写（第 2 个真实用户）时 | +| **L2** | 每语句基类 `ConnectorMetadata` / 契约方法连接器 override | ❌ 需引入每语句 metadata=迷你 Trino 重构 or 只是更重的 L1；碰铁律 A、影响刚稳定的读热路 | +| **L3** | Trino 式每事务 metadata 全重构 | ❌ 远期、跨切、明确碰铁律 A、需独立立项 + 单独签字 | + +--- + +## 5. 将来触发点：paimon 加写（证据） + +- **paimon 的"写未迁移"是真实的、被推迟的路线项**：老的 JNI-writer INSERT 写栈（commit `6c6c9a4b6cd`：`datasource/paimon/PaimonTransaction`、`planner/PaimonTableSink`、`nereids/.../PaimonInsertExecutor`、`transaction/PaimonTransactionManager` + BE `paimon_table_sink_operator`/`PaimonJniWriter`）在**删旧代码期被整体删除、未搬进新 SPI 连接器**。将来的 paimon 写会在写 SPI 上**全新构建**，且 UPDATE/DELETE 是真正新增（老实现只有 INSERT）。 +- **paimon 现在就是"iceberg 改造前"的形状**：胖句柄 `PaimonTableHandle.paimonTable` + 多加载 seam（4 文件，所有连接器最多）。**唯一缺写臂。** +- **一旦加行级 UPDATE/DELETE**：长出"写成形 + beginWrite"两条加载臂 + （若 merge-on-read）"扫描填、写读"的删除暂存 → **完整复现 iceberg 3~5 次加载风暴 + 跨臂暂存需求**。届时移植对 paimon**真正有价值**（架构连贯 + 溶掉胖句柄 + 给暂存一个自然的家），且有 iceberg + paimon 两个真实用户来把共享 helper 的接口形状定对。 + +--- + +## 6. 共享 helper 落点（若将来做 L1） + +- **家 = `fe-connector-api`**（`ConnectorStatementScope`/`ConnectorSession` 已在此；所有连接器都依赖它；是依赖图最低公共祖先；**连接器侧、不碰铁律 A**）。`fe-connector-spi`/`fe-connector-metastore-api` 在其之上，只依赖 api 的连接器看不见，排除。 +- **签名**（一个真原语 + 一个键约定便利）： + ```java + static T computeIfAbsent(ConnectorSession s, String key, Supplier loader) { + return s == null ? loader.get() : s.getStatementScope().computeIfAbsent(key, loader); + } + static T sharedTable(ConnectorSession s, String typePrefix, String db, String tbl, Supplier loader) { + if (s == null) return loader.get(); + String key = typePrefix + ":" + s.getCatalogId() + ":" + db + ":" + tbl + ":" + s.getQueryId(); + return s.getStatementScope().computeIfAbsent(key, loader); + } + ``` +- **iceberg 迁移代价 = 6 处调用 + 删 `IcebergStatementScope`(85 行)**：`sharedTable` 4 处（`IcebergConnectorMetadata:613` 读 / `IcebergConnectorTransaction:211` beginWrite / `IcebergWritePlanProvider:697` 写成形 / `IcebergScanPlanProvider:2274` 扫描）；`rewritableDeleteSupply` 2 处（`IcebergScanPlanProvider:674` 填 / `IcebergWritePlanProvider:193` 读）。 +- **注意**：删除暂存那半只能**结构性**泛化（值类型 `Map>` 是 iceberg thrift、"NONE 不桥接→v3 fail-loud"语义是 iceberg 专属）——泛化时把 iceberg 专属键串 + fail-loud 守卫留在 iceberg 调用点即可。**只有 table-share 那半是真连接器中性。** + +--- + +## 7. 下一步：下个 session 讨论"重构成 Trino 架构"（L2/L3）—— 预备材料 + +> 用户 2026-07-19 指示：本轮先记文档；**重构成 Trino 架构的问题下个 session 专题讨论**。以下是给那次讨论的种子，避免从零起。 + +### 7.1 Trino 架构到底要引入什么（对照 §4.2/§4.3） +1. **每语句/每事务 metadata 实例**：把 `*ConnectorMetadata` 从"每 catalog 共享单例"改成"每语句（或每事务）新建、由引擎 memoize+销毁"。 +2. **一个 span 生命周期管理器**（对应 Trino `TransactionManager`）：即便裸 SELECT 也要有一个活满整条语句的 metadata 宿主；Doris 现成的 span 宿主是 `StatementContext`。 +3. **planner 改造**：Nereids/规划各阶段改成"经 span 取当前语句的 metadata 实例"，而非直接调共享连接器。 +4. **连接器改造**：把表/schema/统计缓存从"跨查询共享缓存 + transient 胖句柄"迁到"挂 per-statement metadata 实例"。 + +### 7.2 必须先摆平的硬冲突（讨论要点） +- **铁律 A（删旧代码期 fe-core 只减不增）**：上述 1/2/3 几乎全是 fe-core 净增 + planner 改造。→ 讨论：是否等删旧代码期结束再做？是否需要用户单独签字放行？ +- **读热路径刚稳定**（PERF-01~06、PERF-11）：per-statement metadata 会重排读热缝，回归面大。 +- **凭证/多租户语义**：per-statement 实例天然按语句隔离（=按用户），能顺带解决"跨查询缓存对 session=user/vended 关闭"的历史包袱——这是 L3 相对现状的**真正架构收益**，值得作为讨论的正面理由。 +- **收益 vs 成本的诚实定位**：现状"连接器向上够 span"已拿到 90% 的去重收益；L3 的增量主要是**架构连贯 + 多租户缓存归一 + 为将来大量写连接器铺路**，而非单点性能。要不要为这些付"跨切重构 + 独立项目"的价，是用户的战略决定。 + +### 7.3 建议的讨论产出 +- 一份"Doris 每语句/每事务 metadata 重构"的**可行性 + 分期**设计（可否分"先 span 宿主、后逐连接器迁移、最后退役共享单例"三期落地，避免一次性大爆炸）。 +- 明确触发条件（如：删旧代码期结束 / 写连接器数量到某阈值）。 + +--- + +## 8. 参考 + +- 蓝本权威设计：`plan-doc/perf-hotpath-iceberg/designs/FIX-PERF-07-unified-per-statement-table-owner-{design,summary}.md`。 +- 代码范本：`fe-connector-iceberg/.../IcebergStatementScope.java`（commit `ea7fd1f6e7a`）；地基 commit `97bdcd6bdbe`。 +- 架构记忆：`iceberg-table-resolution-cache-scoping`（缓存作用域纪律 + Trino 协同 + "全高度留远期"）。 +- 本轮调研原始返回（每 agent 完整结论）：workflow journal + `.../subagents/workflows/wf_e89cf92e-ff3/journal.jsonl`（逐连接器 recon+对抗复核）、 + `.../subagents/workflows/wf_4802a3d2-1c9/journal.jsonl`（placement / onboarding / Trino-altitude 三专项）。 +- 公开 tracking issue：apache/doris#65185。 diff --git a/plan-doc/per-statement-table-owner-port/progress.md b/plan-doc/per-statement-table-owner-port/progress.md index 58bdddb3ad8400..f4fe70709c2307 100644 --- a/plan-doc/per-statement-table-owner-port/progress.md +++ b/plan-doc/per-statement-table-owner-port/progress.md @@ -11,3 +11,18 @@ - 初摸候选（grep）：只有 iceberg 用了该 SPI；`loadTable`/`getTable` 触点 paimon 4 / hive 3 / hudi 1 / maxcompute·es·jdbc·trino 0。→ 候选 = paimon(高)/hive-hms(中-高)/hudi(中)；读-only 四家大概率排除，待复核。 - 落地文件：`README.md`（用途 + 已就位地基 + 连接器侧 5 步模板 + 候选表 + 单连接器立项流程 + 铁律）、`tasklist.md`（PORT-01~04 全待 recon）、`HANDOFF.md`（下一步 = 确认范围 + 逐候选 recon）、`progress.md`（本文件）、`designs/`（空）。 - **未动任何产品代码**。**下一步**：见 HANDOFF —— 下个 session 起步先与用户确认范围/顺序，再对第一个连接器 recon。 + +## 2026-07-19 — session 1：逐连接器 recon + 架构统一性调研（结论：全部 🔬 + 定高度 L0） + +- **逐连接器 recon（recon + 独立对抗复核，双签，多 agent workflow）**：结论 = **没有一个连接器现在值得移植**。 + - paimon → 🔬 **将来候选**：写未迁移（只读）、加载已被 transient 胖句柄 + SDK CachingCatalog 压到≈1；触发点=加行级 UPDATE/DELETE。 + - hive/hms → 🔬 不必做：仅 INSERT/OVERWRITE、跨查询 `CachingHmsClient.tableCache` 已兜（作用域超集）；网关只做 1 次探测加载后委派兄弟（兄弟自带作用域）。 + - hudi → 🔬 不必做：只读；唯一读侧成本=未缓存 metaClient 重建，形状不对/可变/鉴权错配。 + - maxcompute·es·jdbc·trino → 🔬 排除：均无行级 DML；trino 读侧重解析最贵=另立议题占位。 + - **核心洞察**：iceberg 独特在它是唯一迁移了行级写的连接器（行级写才有多臂重复加载 + 跨臂暂存）。 +- **架构统一性专项调研（3 agent：placement / onboarding / Trino-altitude）**： + - **统一接口已存在** = 中性 `getStatementScope()` + `ConnectorStatementScope`（新连接器天生继承）。 + - **Trino 模型不可直接移植**：Trino 靠"引擎自持每事务 metadata 生命周期"，Doris 连接器是共享单例、session 一条语句重建~26 次，唯一 span 宿主=`StatementContext`；现有 SPI 已是最可移植高度。 + - **高度分级 L0/L1/L2/L3**；**paimon-加写=L1 抽共享 helper 的正确触发点**（有 2 个真实用户）；共享 helper 落 `fe-connector-api`（不碰铁律 A）。 +- **用户拍板**：先把结论记成**单独文档** → `designs/recon-findings-and-trino-refactor-groundwork.md`；**重构成 Trino 架构（L2/L3）留下个 session 专题讨论**（该文档 §7 已备预备材料）。 +- **未动任何产品代码**（纯 plan-doc）。**下一步**：见 HANDOFF —— 下个 session = 讨论"重构成 Trino 架构"的可行性/分期/与铁律 A 的取舍。 diff --git a/plan-doc/per-statement-table-owner-port/tasklist.md b/plan-doc/per-statement-table-owner-port/tasklist.md index a6e0435e57f4e6..409b1674025072 100644 --- a/plan-doc/per-statement-table-owner-port/tasklist.md +++ b/plan-doc/per-statement-table-owner-port/tasklist.md @@ -11,10 +11,10 @@ | ID | 连接器 | 候选度 | 状态 | 备注 | commit | |---|---|---|---|---|---| -| PORT-01 | paimon | 高 | ⏳ 待 recon | loadTable 触点最多(4)；写路径结构待确认 | | -| PORT-02 | hive / hms | 中-高 | ⏳ 待 recon | plain-hive 读写活；hms 网关委派 sibling(休眠)——网关按 handle 选 provider 的特殊性要单独设计 | | -| PORT-03 | hudi | 中 | ⏳ 待 recon | 读为主(MTMV)；写臂/多载程度待确认 | | -| PORT-04 | maxcompute / es / jdbc / trino | 低 | ⏳ 待判定 | 0 个 metastore loadTable fan-out；大概率**排除**，下 session 确认后标 🔬 | | +| PORT-01 | paimon | 高 | 🔬 复核判暂不做（**将来候选**） | recon 双签：写未迁移→只读，DML=0；加载已被 transient 胖句柄+SDK CachingCatalog 压到≈1。**触发点=paimon 加行级 UPDATE/DELETE**（届时复现风暴，且是抽共享 helper 的正确时机）。详见 designs 复核结论文档 §2/§5 | | +| PORT-02 | hive / hms | 中-高 | 🔬 复核判不必做 | recon 双签：仅 INSERT/OVERWRITE 无行级写；跨查询 `CachingHmsClient.tableCache` 已把加载压到≈0~1（作用域的超集）；无胖句柄/暂存；网关只做 1 次探测加载后委派兄弟，兄弟自带作用域 | | +| PORT-03 | hudi | 中 | 🔬 复核判不必做 | recon 双签：Doris 侧只读（写全拒）；5 步 4 步空操作；唯一读侧成本=未缓存 metaClient 重建，形状不对+可变+鉴权上下文错配，非本模式 | | +| PORT-04 | maxcompute / es / jdbc / trino | 低 | 🔬 排除 | recon 双签：均无行级 DML（es/trino 只读、jdbc/maxcompute 仅 INSERT-append）；无暂存/胖句柄。**trino 读侧重解析最贵**=另立"读侧去重"议题的占位，不属本模板 | | --- @@ -23,3 +23,10 @@ - iceberg 本身**不在本清单**（已由 PERF-07 完成，见 `plan-doc/perf-hotpath-iceberg/`）。 - 各连接器完成后建各自 `designs/PORT--{design,summary}.md`（不预建）。 - **不立项**的连接器（复核判不必做）标 🔬 + 一句原因，保留占位。 + +## 本轮结论（2026-07-19 · recon 全部完成） + +- **四项全部 🔬**：无连接器现在值得移植——iceberg 独特在它是**唯一迁移了行级写（DELETE/MERGE）**的连接器，别的连接器只读/仅追加写，没有多臂重复加载风暴，现有跨查询缓存又已把加载压到≈1。 +- **统一接口标准已存在**：中性 SPI `getStatementScope()` + `ConnectorStatementScope`（新连接器天生继承）。缺的只是"更强的强制/便利外壳"，现在做=单用户过早抽象。 +- **推荐高度 L0**（写下约定 + 登记 paimon-加写触发点，生产逻辑零改动）；L1（抽共享 helper）留到 paimon 加写；**L2/L3（Trino 式重构）下个 session 专题讨论**。 +- 全部依据：`designs/recon-findings-and-trino-refactor-groundwork.md`。 From c4959a9de5cf18b6d69a4109713a53fcd2a6333b Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 15:27:03 +0800 Subject: [PATCH 29/47] [feat](catalog) fe-core: engine-side per-statement metadata funnel (foundation) Foundation for the per-statement ConnectorMetadata redesign: an engine-owned funnel that memoizes one ConnectorMetadata per (statement, catalog) and can close it deterministically at statement end. Read-side rerouting, close wiring, HMS sibling, write sharing, and cache isolation land in later commits. - ConnectorStatementScope (SPI): add getOrCreateMetadata(key, factory) typed memo over computeIfAbsent + closeAll() default no-op. NONE runs the factory every call (byte-identical to today). - ConnectorStatementScopeImpl: idempotent best-effort closeAll() closing each AutoCloseable value once (close-once guard, log-and-continue). - PluginDrivenMetadata: the single funnel - get(session, connector) memoizes connector.getMetadata(session) on the session's per-statement scope, keyed by catalog id. - Tests: memo-once-per-statement, NONE-each-call, cross-catalog isolation, close-once idempotency. No production seam routes through the funnel yet; this commit is byte-neutral and independently compilable. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../api/ConnectorStatementScope.java | 22 ++++ .../ConnectorStatementScopeImpl.java | 32 +++++ .../plugin/PluginDrivenMetadata.java | 56 ++++++++ .../ConnectorStatementScopeTest.java | 20 +++ .../plugin/PluginDrivenMetadataTest.java | 122 ++++++++++++++++++ 5 files changed, 252 insertions(+) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenMetadata.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenMetadataTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatementScope.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatementScope.java index d12d2b45c15485..4c821a4c4de674 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatementScope.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatementScope.java @@ -44,6 +44,28 @@ public interface ConnectorStatementScope { */ T computeIfAbsent(String key, Supplier loader); + /** + * Typed convenience over {@link #computeIfAbsent} for the ONE {@link ConnectorMetadata} a statement uses per + * {@code key}. The engine's metadata funnel builds {@code key} (as {@code "metadata:" + catalogId}, plus the + * owning connector's label for a heterogeneous gateway) and passes a {@code factory} that calls + * {@code Connector#getMetadata(session)}; every read / scan / DDL / MVCC resolver of the statement then shares + * the single memoized instance and the factory runs at most once per statement. Under {@link #NONE} the + * factory runs on every call (byte-identical to building metadata every time). + */ + default ConnectorMetadata getOrCreateMetadata(String key, Supplier factory) { + return computeIfAbsent(key, factory); + } + + /** + * Deterministically closes, at statement end, every value this scope holds that is {@link AutoCloseable} (a + * memoized {@link ConnectorMetadata} is, via {@link java.io.Closeable}). Best-effort and log-and-continue: a + * failure closing one value does not abort the rest. MUST be idempotent (close-once) in implementations — the + * engine fires it from more than one locus (the query-finish callback, and a reused prepared statement's + * per-execution reset). The default is a no-op, so {@link #NONE} — which memoizes nothing — stays inert. + */ + default void closeAll() { + } + /** The no-op scope: never caches; each call invokes the loader (offline / no-context / tests). */ ConnectorStatementScope NONE = new ConnectorStatementScope() { @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java index 6d40721b0edaba..238d1d18d97ef9 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorStatementScopeImpl.java @@ -19,6 +19,9 @@ import org.apache.doris.connector.api.ConnectorStatementScope; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + import java.util.concurrent.ConcurrentHashMap; import java.util.function.Supplier; @@ -35,11 +38,40 @@ */ public class ConnectorStatementScopeImpl implements ConnectorStatementScope { + private static final Logger LOG = LogManager.getLogger(ConnectorStatementScopeImpl.class); + private final ConcurrentHashMap cache = new ConcurrentHashMap<>(); + private boolean closed; @Override @SuppressWarnings("unchecked") public T computeIfAbsent(String key, Supplier loader) { return (T) cache.computeIfAbsent(key, k -> loader.get()); } + + /** + * Closes every {@link AutoCloseable} value once, at statement end. Idempotent: guarded by {@code closed} so a + * second trigger (the query-finish callback vs. a reused prepared statement's per-execution reset) is a + * harmless no-op and no value is double-closed. Best-effort per value — a failure closing one does not abort + * the rest — mirroring the isolation of the engine's query-finish callback registry. Runs after the scan + * off-thread pumps have quiesced, so it does not race a concurrent {@code computeIfAbsent}. + */ + @Override + public synchronized void closeAll() { + if (closed) { + return; + } + closed = true; + for (Object value : cache.values()) { + if (value instanceof AutoCloseable) { + try { + ((AutoCloseable) value).close(); + } catch (Exception e) { + LOG.warn("failed to close per-statement scope value of type {}; continuing", + value.getClass().getName(), e); + } + } + } + cache.clear(); + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenMetadata.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenMetadata.java new file mode 100644 index 00000000000000..db5689f5a69984 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenMetadata.java @@ -0,0 +1,56 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.plugin; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; + +/** + * The single engine-side funnel through which every metadata acquisition in a statement flows, so that a + * statement uses exactly ONE {@link ConnectorMetadata} instance per catalog: it memoizes the result of + * {@code connector.getMetadata(session)} on the session's per-statement {@link ConnectorStatementScope} and hands + * every read / scan / DDL / MVCC resolver the same instance, closed deterministically at statement end. + * + *

    Under {@link ConnectorStatementScope#NONE} (offline / no live statement) the scope memoizes nothing, so the + * factory runs on every call — {@code connector.getMetadata(session)} per call, byte-identical to today's behavior. + * The {@link ConnectorSession} is still built and passed to metadata operations as before; only the + * {@code getMetadata} result is memoized here, never the session.

    + * + *

    This class is the ONLY place in fe-core allowed to call {@code Connector#getMetadata} directly (an arch gate + * enforces that every other seam routes through here). A heterogeneous gateway connector that fans a foreign + * handle out to a sibling connector keys the memo by the owning connector as well as the catalog id; that + * sibling-aware entry is added when the gateway is wired.

    + */ +public final class PluginDrivenMetadata { + + private PluginDrivenMetadata() { + } + + /** + * Returns the statement's single memoized {@link ConnectorMetadata} for {@code connector}, building it via + * {@code connector.getMetadata(session)} on first use and reusing it for the rest of the statement. Keyed by + * the session's catalog id, so a cross-catalog statement resolves each catalog independently. + */ + public static ConnectorMetadata get(ConnectorSession session, Connector connector) { + ConnectorStatementScope scope = session.getStatementScope(); + String key = "metadata:" + session.getCatalogId(); + return scope.getOrCreateMetadata(key, () -> connector.getMetadata(session)); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java index a4a6e3726c9e73..ce640d9c582d09 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java @@ -92,4 +92,24 @@ public void statementContextScopeIsStableThenResetsForReusedPreparedContext() { Assertions.assertNotSame(memoized, s2.computeIfAbsent("k", Object::new), "a prior execution's memoized value must not leak into the next execution"); } + + @Test + public void closeAllClosesCloseableValuesOnceAndIgnoresPlainValues() throws Exception { + // closeAll() closes every AutoCloseable value the statement memoized (a ConnectorMetadata is Closeable) + // and leaves non-closeable values (the shared table object, the scan->write delete-supply map) untouched. + // It must be idempotent: the engine fires it from more than one locus (the query-finish callback + a reused + // prepared statement's per-execution reset), so a second call must not double-close. + // MUTATION: dropping the close-once guard -> closes==2 -> red. + ConnectorStatementScope scope = new ConnectorStatementScopeImpl(); + AtomicInteger closes = new AtomicInteger(); + AutoCloseable closeable = () -> closes.incrementAndGet(); + scope.computeIfAbsent("closeable", () -> closeable); + scope.computeIfAbsent("plain", Object::new); // a non-closeable value must be ignored, not crash + + scope.closeAll(); + scope.closeAll(); + + Assertions.assertEquals(1, closes.get(), + "each closeable value is closed exactly once across repeated closeAll (idempotent)"); + } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenMetadataTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenMetadataTest.java new file mode 100644 index 00000000000000..5849a6a329657a --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenMetadataTest.java @@ -0,0 +1,122 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.plugin; + +import org.apache.doris.connector.ConnectorStatementScopeImpl; +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.concurrent.atomic.AtomicInteger; + +/** + * Pins the engine-side metadata funnel {@link PluginDrivenMetadata}: within one statement it hands every caller + * the SAME memoized {@link ConnectorMetadata} per catalog (so read / scan / DDL / MVCC share one), isolates + * distinct catalogs, runs the factory on every call under {@link ConnectorStatementScope#NONE} (offline / + * byte-identical to today), and lets the scope close the memoized instance exactly once at statement end. + */ +public class PluginDrivenMetadataTest { + + private static ConnectorSession session(long catalogId, ConnectorStatementScope scope) { + ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getCatalogId()).thenReturn(catalogId); + Mockito.when(session.getStatementScope()).thenReturn(scope); + return session; + } + + private static Connector countingConnector(AtomicInteger builds) { + Connector connector = Mockito.mock(Connector.class); + Mockito.when(connector.getMetadata(Mockito.any())).thenAnswer(inv -> { + builds.incrementAndGet(); + return Mockito.mock(ConnectorMetadata.class); + }); + return connector; + } + + @Test + public void memoizesOneMetadataPerStatement() { + // The read/scan/DDL/MVCC resolvers of one statement all route through the funnel and must collapse onto a + // single getMetadata build and share the one instance. MUTATION: not memoizing -> two builds / two + // instances -> red. + ConnectorStatementScope scope = new ConnectorStatementScopeImpl(); + ConnectorSession session = session(7L, scope); + AtomicInteger builds = new AtomicInteger(); + Connector connector = countingConnector(builds); + + ConnectorMetadata first = PluginDrivenMetadata.get(session, connector); + ConnectorMetadata second = PluginDrivenMetadata.get(session, connector); + + Assertions.assertSame(first, second, "one statement + one catalog -> one shared metadata instance"); + Assertions.assertEquals(1, builds.get(), "connector.getMetadata is built once per statement"); + } + + @Test + public void noneBuildsMetadataEachCall() { + // No live statement scope (offline / tests / no ConnectContext): the funnel must degrade to today's + // behavior -> a fresh getMetadata every call, byte-identical to pre-funnel. MUTATION: NONE memoizing -> + // one build -> red. + ConnectorSession session = session(7L, ConnectorStatementScope.NONE); + AtomicInteger builds = new AtomicInteger(); + Connector connector = countingConnector(builds); + + ConnectorMetadata a = PluginDrivenMetadata.get(session, connector); + ConnectorMetadata b = PluginDrivenMetadata.get(session, connector); + + Assertions.assertNotSame(a, b, "NONE memoizes nothing -> a fresh metadata each call"); + Assertions.assertEquals(2, builds.get(), "NONE runs the factory (getMetadata) every call"); + } + + @Test + public void differentCatalogIdIsolates() { + // A cross-catalog statement (e.g. a MERGE reading two catalogs) resolves each catalog's metadata + // independently, because the memo key carries the catalog id. MUTATION: dropping the catalog id from the + // key -> the two catalogs collide onto one instance -> red. + ConnectorStatementScope scope = new ConnectorStatementScopeImpl(); + AtomicInteger builds = new AtomicInteger(); + Connector connector = countingConnector(builds); + + ConnectorMetadata c1 = PluginDrivenMetadata.get(session(1L, scope), connector); + ConnectorMetadata c2 = PluginDrivenMetadata.get(session(2L, scope), connector); + + Assertions.assertNotSame(c1, c2, "distinct catalogs are isolated"); + Assertions.assertEquals(2, builds.get(), "one build per distinct catalog"); + } + + @Test + public void closeAllClosesTheMemoizedMetadataOnce() throws Exception { + // The statement's one memoized ConnectorMetadata (which is Closeable) is closed deterministically at + // statement end, exactly once even though the engine fires closeAll from more than one locus (query-finish + // callback + prepared-statement reset). MUTATION: closeAll not idempotent -> close() called twice -> red. + ConnectorStatementScopeImpl scope = new ConnectorStatementScopeImpl(); + ConnectorSession session = session(7L, scope); + ConnectorMetadata md = Mockito.mock(ConnectorMetadata.class); + Connector connector = Mockito.mock(Connector.class); + Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(md); + + PluginDrivenMetadata.get(session, connector); + scope.closeAll(); + scope.closeAll(); + + Mockito.verify(md, Mockito.times(1)).close(); + } +} From 6d35335ca39e02183c10a6aad115dec165052bfa Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 16:19:48 +0800 Subject: [PATCH 30/47] [feat](catalog) fe-core: deterministic close of the per-statement metadata scope (close wiring) Wire the per-statement ConnectorStatementScope to be closed deterministically at statement end, so a later commit that gives ConnectorMetadata.close() real work (releasing FileIO / tables) releases it exactly once, not on GC. P1 close is still a no-op, so this commit is behavior-neutral; it establishes the lifecycle. Two-tier close (a single register-at-scope-creation point was rejected: it would leak the callback for DDL / SHOW / DESCRIBE / EXPLAIN / foreground ANALYZE, which create a scope via Command.run but never reach unregisterQuery, and the callback registry has no TTL): - PRIMARY (coordinated scans/writes/cursor-fetch/internal): getSplits registers scope::closeAll on the existing query-finish callback, same query-id key as the read-transaction release beside it. Object-capture; skip NONE. Fires after off-thread pump quiescence; no registry leak. - FALLBACK (non-coordinated Command statements): StatementContext.close() closes the scope, guarded by isReturnResultFromLocal so arrow-flight defers to its own query-finish close. Runs in ConnectProcessor's per-statement finally (direct) and a new finally in proxyExecute (master-side forwarded DDL/SHOW). - resetConnectorStatementScope() closes-before-null, and each retry attempt resets, so a reused StatementContext (prepared EXECUTE / retry) never memoizes into an already-closed scope whose values would never close. closeAll is idempotent, so the dual primary+fallback triggers on a coordinated local query close the scope exactly once. Tests: reset-closes-before-drop, close()-closes-locally, close()-defers-for-async. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../datasource/scan/PluginDrivenScanNode.java | 13 +++++ .../doris/nereids/StatementContext.java | 25 ++++++-- .../org/apache/doris/qe/ConnectProcessor.java | 10 ++++ .../org/apache/doris/qe/StmtExecutor.java | 7 +++ .../ConnectorStatementScopeTest.java | 58 +++++++++++++++++++ 5 files changed, 109 insertions(+), 4 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index 80d9c3e381a4d2..adb30c8fed359a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -33,6 +33,7 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.PassthroughQueryTableHandle; @@ -1207,6 +1208,18 @@ public List getSplits(int numBackends) throws UserException { QeProcessorImpl.INSTANCE.registerQueryFinishCallback(readTxnQueryId, buildReadTransactionReleaseCallback(scanProvider, readTxnQueryId)); + // Deterministic close of the per-statement metadata scope, on the SAME query-finish hook and the SAME + // query-id key as the read-transaction release above. This is the PRIMARY close: getSplits runs only for + // coordinated scans, all of which reach unregisterQuery, so it fires after off-thread pump quiescence and + // leaves no dangling registry entry. Object-capture (scope::closeAll binds THIS scope instance) so a retry + // / prepared-EXECUTE that swaps the StatementContext field can never let this callback touch a successor + // scope. Skip NONE (off-thread / no-ConnectContext builds carry NONE and hold nothing to close). Non-scan + // statements (DDL / SHOW / EXPLAIN via Command.run) never reach here and are closed by StatementContext. + ConnectorStatementScope statementScope = connectorSession.getStatementScope(); + if (statementScope != ConnectorStatementScope.NONE) { + QeProcessorImpl.INSTANCE.registerQueryFinishCallback(readTxnQueryId, statementScope::closeAll); + } + // Push the Nereids partition-pruning result down to the connector so the read session // covers only the surviving partitions. A pruned-to-zero set means no data to read, // mirroring legacy MaxComputeScanNode.getSplits()'s empty-selection short-circuit — UNLESS the diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java index 9f1e5a47d69c24..5541d8ae57dee4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java @@ -661,12 +661,18 @@ public synchronized ConnectorStatementScope getOrCreateConnectorStatementScope() } /** - * Drops the connector scope so the next statement execution starts fresh. Prepared-statement EXECUTE - * reuses one StatementContext across executions (see - * {@link org.apache.doris.nereids.trees.plans.commands.ExecuteCommand}); this is called there, beside - * the other per-execution resets, so a prior execution's cached tables/state never leak into the next. + * Closes (deterministically) then drops the connector scope so the next statement execution starts fresh. + * Prepared-statement EXECUTE reuses one StatementContext across executions (see + * {@link org.apache.doris.nereids.trees.plans.commands.ExecuteCommand}) and a retry reuses it across attempts; + * this is called at the start of each, so a prior execution's / attempt's cached tables and closeable state + * are released (closeAll is idempotent — a no-op if the query-finish callback already closed it) and never + * leak into the next. Callers invoke this only after the prior execution/attempt has finished (no off-thread + * pump still running), so close-before-drop is safe. */ public synchronized void resetConnectorStatementScope() { + if (this.connectorStatementScope != null) { + this.connectorStatementScope.closeAll(); + } this.connectorStatementScope = null; } @@ -978,6 +984,17 @@ protected void finalize() throws Throwable { @Override public void close() { releasePlannerResources(); + // Fallback deterministic close of the per-statement connector scope, for statements that never reach the + // query-finish callback: external DDL / SHOW / DESCRIBE / EXPLAIN / foreground ANALYZE run via Command.run + // with no coordinator, so PluginDrivenScanNode.getSplits never registers a primary close for them. close() + // runs in ConnectProcessor's per-statement finally (direct connection) and the forwarded-request finally + // (master side), so it covers them. Guarded by isReturnResultFromLocal so an arrow-flight statement (which + // returns results asynchronously and defers cleanup to its own query-finish close) is not closed early + // here. Idempotent (closeAll is close-once), so a coordinated statement whose scope the primary already + // closed is a no-op. Command statements have no off-thread scan pump, so close-after-use holds. + if (connectorStatementScope != null && connectContext != null && connectContext.isReturnResultFromLocal()) { + connectorStatementScope.closeAll(); + } } public List getPlaceholders() { diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java index 72bbf95a372ce9..c83397f58dc0e7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java @@ -784,6 +784,16 @@ public TMasterOpResult proxyExecute(TMasterOpRequest request) throws TException // If reach here, maybe Doris bug. LOG.warn("Process one query failed because unknown reason: ", e); ctx.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, "Unexpected exception: " + e.getMessage()); + } finally { + // Master-side forwarded statements (redirect DDL / SHOW) execute via Command.run and never reach + // unregisterQuery, so their per-statement connector scope has no query-finish trigger. Close it here + // via StatementContext.close() (the same per-statement close the direct-connection path runs in its + // finally). Idempotent: a coordinated forwarded query's scope is already closed by its query-finish + // callback. These statements run synchronously with no off-thread scan pump, so close-after-use holds. + StatementContext forwardedStatementContext = ctx.getStatementContext(); + if (forwardedStatementContext != null) { + forwardedStatementContext.close(); + } } // no matter the master execute success or fail, the master must transfer the result to follower // and tell the follower the current journalID. diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java index 92f698b1950b36..7bad37645b7cae 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java @@ -1069,6 +1069,13 @@ private void handleQueryWithRetry(TUniqueId queryId) throws Exception { DebugUtil.printId(queryId), i, DebugUtil.printId(newQueryId)); context.setQueryId(newQueryId); context.setNeedRegenerateInstanceId(newQueryId); + // Each retry attempt gets a fresh per-statement connector scope. The previous attempt's scope + // was closed by its own query-finish callback (registered under the previous query id), so + // reusing it would memoize into an already-closed scope whose values then never close. Reset + // closes (idempotent) and drops it; this attempt builds a fresh one under the new query id. + if (context.getStatementContext() != null) { + context.getStatementContext().resetConnectorStatementScope(); + } if (Config.isCloudMode()) { // sleep random millis [1000, 1500] ms // in the begining of retryTime/2 diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java index ce640d9c582d09..14d11d6d0462d0 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java @@ -19,9 +19,11 @@ import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.nereids.StatementContext; +import org.apache.doris.qe.ConnectContext; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.mockito.Mockito; import java.util.concurrent.atomic.AtomicInteger; @@ -112,4 +114,60 @@ public void closeAllClosesCloseableValuesOnceAndIgnoresPlainValues() throws Exce Assertions.assertEquals(1, closes.get(), "each closeable value is closed exactly once across repeated closeAll (idempotent)"); } + + @Test + public void resetClosesTheDroppedScopeBeforeStartingFresh() { + // A prepared EXECUTE / retry reuses one StatementContext and calls resetConnectorStatementScope() at the + // start of each execution/attempt. Reset must CLOSE the outgoing scope's closeable values (else the prior + // execution's tables/FileIO leak once close() does real work) and then drop it so the next starts fresh. + // MUTATION: reset only nulling (not closing) -> closes==0 -> red. + StatementContext ctx = new StatementContext(); + ConnectorStatementScope s1 = ctx.getOrCreateConnectorStatementScope(); + AtomicInteger closes = new AtomicInteger(); + AutoCloseable closeable = () -> closes.incrementAndGet(); + s1.computeIfAbsent("k", () -> closeable); + + ctx.resetConnectorStatementScope(); + + Assertions.assertEquals(1, closes.get(), "reset closes the dropped scope's closeable values before dropping it"); + Assertions.assertNotSame(s1, ctx.getOrCreateConnectorStatementScope(), + "reset drops the scope so the next execution/attempt starts fresh"); + } + + @Test + public void closeClosesScopeWhenReturningResultLocally() { + // A statement that never reaches the query-finish callback (external DDL / SHOW via Command.run) is + // closed by StatementContext.close(), the fallback ConnectProcessor runs in its per-statement finally. + // It fires only when the result is produced locally. MUTATION: close() not closing the scope -> red. + ConnectContext connectContext = Mockito.mock(ConnectContext.class); + Mockito.when(connectContext.isReturnResultFromLocal()).thenReturn(true); + StatementContext ctx = new StatementContext(connectContext, null); + ConnectorStatementScope scope = ctx.getOrCreateConnectorStatementScope(); + AtomicInteger closes = new AtomicInteger(); + AutoCloseable closeable = () -> closes.incrementAndGet(); + scope.computeIfAbsent("k", () -> closeable); + + ctx.close(); + + Assertions.assertEquals(1, closes.get(), "close() closes the scope for a locally-returned statement"); + } + + @Test + public void closeDefersScopeCloseForAsyncResult() { + // An arrow-flight statement returns results asynchronously (isReturnResultFromLocal == false) and defers + // scope cleanup to its own query-finish close; StatementContext.close() must NOT close it early here, or an + // in-flight fetch would touch a closed scope. MUTATION: dropping the guard -> closes==1 -> red. + ConnectContext connectContext = Mockito.mock(ConnectContext.class); + Mockito.when(connectContext.isReturnResultFromLocal()).thenReturn(false); + StatementContext ctx = new StatementContext(connectContext, null); + ConnectorStatementScope scope = ctx.getOrCreateConnectorStatementScope(); + AtomicInteger closes = new AtomicInteger(); + AutoCloseable closeable = () -> closes.incrementAndGet(); + scope.computeIfAbsent("k", () -> closeable); + + ctx.close(); + + Assertions.assertEquals(0, closes.get(), + "close() defers to the query-finish callback for async (arrow-flight) results"); + } } From 4f6ff211256eb46e3b916396c6d046745fe4b877 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 16:23:17 +0800 Subject: [PATCH 31/47] [doc](catalog) per-statement metadata redesign: phasing decisions + STEP 1 (C1 foundation + C2 close wiring) - expanded-scope-phasing-and-security-decisions.md: user decisions (phased, not big-bang; the "propagate the instance to background threads" ask was refuted by recon and corrected to read-through; cache isolation confirmed as a REAL security fix since list != load) + verified 3-area findings + STEP 1-4 sequencing + C1/C2 implementation progress + close-wiring residual risks. - trino-parity-metadata-redesign-design.md, P1-implementation-design.md: target architecture + seam-by-seam blueprint (from the prior session). P1-design section 4 corrected: the "register at scope creation" close plan was found to leak (DDL/SHOW/EXPLAIN/ANALYZE via Command.run create a scope but never reach unregisterQuery, and the callback registry has no TTL); replaced with the two-tier close landed in the code commits. - HANDOFF / progress / tasklist: C1 (5b7312f9d1f) + C2 (12f3e95239b) marked done; next session = STEP 1 C3 (read-side rerouting + background read-through + fix the ANALYZE row-count scope-binding hazard). Docs only. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../per-statement-table-owner-port/HANDOFF.md | 67 ++--- .../designs/P1-implementation-design.md | 180 ++++++++++++++ ...ed-scope-phasing-and-security-decisions.md | 96 ++++++++ .../trino-parity-metadata-redesign-design.md | 228 ++++++++++++++++++ .../progress.md | 36 +++ .../tasklist.md | 17 +- 6 files changed, 594 insertions(+), 30 deletions(-) create mode 100644 plan-doc/per-statement-table-owner-port/designs/P1-implementation-design.md create mode 100644 plan-doc/per-statement-table-owner-port/designs/expanded-scope-phasing-and-security-decisions.md create mode 100644 plan-doc/per-statement-table-owner-port/designs/trino-parity-metadata-redesign-design.md diff --git a/plan-doc/per-statement-table-owner-port/HANDOFF.md b/plan-doc/per-statement-table-owner-port/HANDOFF.md index fba41a006c7d38..06468b43f64919 100644 --- a/plan-doc/per-statement-table-owner-port/HANDOFF.md +++ b/plan-doc/per-statement-table-owner-port/HANDOFF.md @@ -1,49 +1,58 @@ -# 🤝 Session Handoff —— 每语句表加载归属者 · 移植到其它连接器 +# 🤝 Session Handoff —— 每语句表加载归属者 · 移植/重构 > **滚动文档**：每 session 结束覆盖式更新，只留下一个 session 必须的上下文。 > 开场必读顺序、模板、铁律见 [`README.md`](./README.md)；进度见 [`progress.md`](./progress.md)；状态见 [`tasklist.md`](./tasklist.md)。 --- -# 🆕 本轮（2026-07-19 session 1）已完成：**全部连接器 recon + 架构统一性调研 → 结论全部 🔬** +# 🆕 本轮（2026-07-19 session 3）已完成：**定稿分期 + 落地读取键石的 C1 地基 + C2 关闭接线** ## 一句话结果 -- **逐连接器复核（双签）：没有一个连接器现在值得移植**——iceberg 独特在它是**唯一迁移了行级写（DELETE/MERGE）**的连接器；别的连接器只读 / 仅追加写，没有多臂重复加载风暴，现有跨查询缓存又已把加载压到≈1。四项全部标 🔬。 -- **"统一接口标准"其实已存在**（中性 `getStatementScope()` + `ConnectorStatementScope`，新连接器天生继承）；**推荐高度 L0**（写下约定 + 登记触发点，生产逻辑零改动）。 -- **用户拍板**：本轮先把结论记成**单独文档**（已落 `designs/recon-findings-and-trino-refactor-groundwork.md`）；**重构成 Trino 架构（L2/L3）留到下个 session 专题讨论**。 -- **未动任何产品代码**（纯 plan-doc）。 +- 用户拍板扩展范围并定**分期推进**：读写共用、缓存隔离都做，但排序 + 独立提交；后台"传句柄到后台线程"被取证证伪 → 改"读穿 + 修隐患"；缓存"list ≠ load"已确认 → 缓存隔离是**真实安全修复**。定稿见 `designs/expanded-scope-phasing-and-security-decisions.md`。 +- **已实现并提交（均编译通过 + checkstyle 0 + 单测全绿）**： + - **C1 地基** `5b7312f9d1f`：SPI 加 `getOrCreateMetadata`/`closeAll` 默认方法 + `ConnectorStatementScopeImpl.closeAll`(幂等) + 静态漏斗 `PluginDrivenMetadata.get`。字节中性（无生产路径接进漏斗）。 + - **C2 关闭接线** `12f3e95239b`：**两层关闭**（主=getSplits 注册 `scope::closeAll`；兜底=`StatementContext.close` 守卫 closeAll + `proxyExecute` finally；reset closeAll-before-null；retry 每次重置）。P1 关闭仍 no-op，行为中性。 +- 取证 workflow：`wf_8b907b93-e9f`（扩展范围三块）、`wf_9250330b-e81`（关闭接线生命周期，**推翻了原"scope 创建处注册"会泄漏**）。 --- -# ➡️ 下一个 session = **专题讨论"重构成 Trino 架构"（L2/L3）** +# ➡️ 下一个 session = **STEP 1 的 C3：读取侧改道 + 后台读穿纠正（本阶段最大一步，动手前先过清单 + 配对抗复核）** -## 第一件事（先读，别急着给方案） -1. 读 **`designs/recon-findings-and-trino-refactor-groundwork.md`**（本轮全部结论：逐连接器复核 §2、iceberg 为何独特 §3、统一性/Trino 参照/高度分级 §4、paimon 触发点 §5、共享 helper 落点 §6、**Trino 重构预备材料 §7**）。 -2. 读架构记忆 `iceberg-table-resolution-cache-scoping`（缓存作用域纪律 + "全高度留远期"）。 +## 第一件事（先读） +1. 读 `designs/expanded-scope-phasing-and-security-decisions.md`（分期定稿 + §6 C1/C2 进展 + 关闭接线残留风险）。 +2. 读 `designs/P1-implementation-design.md`：§2 的 66 缝改道表（C3 的清单）；§3 扫描节点存字段；**§4 已更正为两层关闭**（C2 已实现，勿再按旧"scope 创建处注册"）。 +3. 读架构记忆 `iceberg-table-resolution-cache-scoping`。 -## 讨论要点（种子，详见结论文档 §7） -- **Trino 到底要引入什么**：每语句/每事务 `ConnectorMetadata` 实例（现状是每 catalog 共享单例）+ 一个 span 生命周期管理器（Doris 现成 span 宿主=`StatementContext`）+ planner 改成经 span 取 metadata + 逐连接器把缓存迁到 per-statement 实例。 -- **必须先摆平的硬冲突**： - - **铁律 A**（删旧代码期 fe-core 只减不增）——L2/L3 几乎全是 fe-core 净增 + planner 改造 → 是否等删旧代码期结束？是否需用户单独签字？ - - 读热路径刚稳定（PERF-01~06/11），per-statement metadata 会重排读热缝、回归面大。 - - **正面收益**：per-statement 实例天然按语句/用户隔离 → 顺带解决"跨查询缓存对 session=user/vended 关闭"的历史包袱（这是 L3 相对现状的真正架构收益）。 -- **建议讨论产出**：一份"Doris 每语句/每事务 metadata 重构"的**可行性 + 分期**设计（先 span 宿主 → 逐连接器迁移 → 退役共享单例，避免一次性大爆炸）+ 明确触发条件。 +## C3 要做的（读取侧改道 + 后台纠正） +- **读/扫描/DDL/MVCC/misc 改道**（~55 处 on-thread）：把 `connector.getMetadata(session)` 改调 `PluginDrivenMetadata.get(session, connector)`。清单见 §2 组 A/B/C/E/misc。 +- **扫描节点存字段**（组 C）：`PluginDrivenScanNode` 加懒字段 `cachedMetadata` + `metadata()`，8 处 per-method 重取改调它（`volatile` 稳态）。 +- **后台读穿纠正**（用户拍板）：7 处跨语句 loader（`listDatabaseNames`/`listTableNamesFromRemote`/`initSchema`/`getColumnStatistic`/`getChunkSizes`/`fetchRowCount`/`MetadataGenerator.dealPluginDrivenCatalog`）**显式建 NONE session 读穿**（`ConnectorSessionBuilder.withStatementScope(NONE)` 或新增 `buildCrossStatementSession()`）；**修 `fetchRowCount` 在 ANALYZE 线程（`AnalysisManager.buildAnalysisJobInfo`）被错误绑定到语句作用域的隐患**。 +- **⚠ 别误改道 `PluginDrivenExternalTable.resolveWriteTargetHandle`**（藏在读文件里的**写专用**点）——标记为写 seam，留 STEP 3。 +- 守门：一语句一表加载计数=1（对照 NONE=N）；跨 catalog/queryId 隔离。 +- **动手前**：先把 §2 改道表逐条核对当前代码行号（可能漂移），列清单给用户过一遍，并计划一轮对抗式复核（clean-room 偏好）。 -## 关键已知（省得重新发现） -- **地基已就位、勿再改**：`ConnectorStatementScope` + `getStatementScope()`(fe-connector-api) + `ConnectorStatementScopeImpl` + `StatementContext` 懒建/重置 + `ConnectorSessionImpl` 构造期捕获 + `ExecuteCommand` 重置(fe-core)。 -- **目前只有 iceberg 用了该 SPI**（`IcebergStatementScope` + `IcebergWritePlanProvider`）。 -- **paimon = 唯一真实的将来 L1 候选**：写未迁移（老 JNI-writer INSERT 写栈在删旧代码期被删、未搬进新连接器）；已有胖句柄 `PaimonTableHandle.paimonTable` + 4 加载 seam；**加行级 UPDATE/DELETE 时才值得移植**，届时抽共享 helper（落 `fe-connector-api`，签名/6 处迁移见结论文档 §6）。 +## 后续步（STEP 2/3/4，详见分期定稿 §4） +- **STEP 2 HMS 兄弟**：键含属主 label + 兄弟 getMetadata **及 beginTransaction** 进漏斗 + 异构网关 e2e。 +- **STEP 3 写入共用**：3a 无状态写点改道（含被标记的 `resolveWriteTargetHandle`）；3b `ConnectorTransaction` 归属上移 `CatalogStatementTransaction`。闸门=读写身份等价 + 保留 hive 起写刷新 + 保留 tx↔session 绑定。纠缠点：iceberg 起写复用表读 `IcebergStatementScope.sharedTable`（P2 计划删）——3b 时定"先接旧 vs 先做 iceberg 最小 P2 前置"。 +- **STEP 4 缓存隔离（安全）**：`getIdentityShardKey()` SPI → iceberg 三投影缓存分片 + fe-core 表结构缓存 bypass(先)/分片(后) + 防漂移门禁；随 STEP 2 属主键覆盖异构网关；越权 e2e + 威胁模型签字。 -## 铁律提醒 -- 地基勿再改；移植（若做）只写连接器侧；暴露地基缺口先停手交 review。 -- L2/L3 碰铁律 A（fe-core 净增）——**属独立立项 + 需用户签字**，不在本"纯连接器侧移植"任务的默认范围内。 -- 作用域跨用户即泄漏——凡有 session=user/凭证语义的连接器，复核共享安全（本轮已确认现有各连接器均目录级单一身份、不泄漏）。 +## 关闭接线残留风险（carry-forward，详见分期定稿 §6） +1. 取消/超时非硬栅栏（`SplitAssignment.stop` 只置标志不 join pump）——既有共担，P1 no-op 下无实害，**关闭做实事前须硬化**。 +2. arrow-flight 异常断连 → 注册表条目留存（无 TTL）——既有共担。 +3. **待确认**：走协调器但从不走 getSplits 的外部目录查询（纯 information_schema / 某元数据 TVF）可能漏关——STEP 3/后续确认。 +4. **🔴 TCCL 自钉扎（硬前置）**：一旦连接器 `close()` 做实事（P2+），主关闭回调 + 兜底两处都须把 TCCL 钉到插件 classloader（连接器 `close()` 自钉扎首选）。 + +## 铁律 / 闸门提醒 +- 用户 2026-07-19 已豁免铁律 A（fe-core 只减不增）——本重构的 fe-core 净增已获授权。仍守：连接器 connector-agnostic（作用域值 Object）；作用域跨用户即泄漏（STEP 4 威胁模型）。 +- **动码前探测并发活动**（git log/status + maven 进程 + 近 90s mtime），发现活跃即停手（`concurrent-sessions-shared-worktree-hazard`）。 --- # 🗂 遗留 / 关联 -- 本轮全部结论：`designs/recon-findings-and-trino-refactor-groundwork.md`。 -- iceberg 蓝本任务空间：`../perf-hotpath-iceberg/`（PERF-07 权威设计/小结）。 +- 分期定稿（现行主线）：`designs/expanded-scope-phasing-and-security-decisions.md`（含 §6 进展 + 残留风险）。 +- STEP 1 蓝图：`designs/P1-implementation-design.md`（§2 改道表；§4 已更正为两层关闭）。 +- 目标架构全景：`designs/trino-parity-metadata-redesign-design.md`。 +- 蓝本 iceberg：`../perf-hotpath-iceberg/`（PERF-07）；代码范本 `IcebergStatementScope`(commit `ea7fd1f6e7a`)+地基 `97bdcd6bdbe`。 - 架构记忆：`iceberg-table-resolution-cache-scoping`。 -- 本轮调研原始返回：workflow journal `wf_e89cf92e-ff3`（逐连接器）+ `wf_4802a3d2-1c9`（统一性三专项），路径见结论文档 §8。 -- e2e 一律留各连接器进 `SPI_READY_TYPES` 切换阶段统一补（对齐 `hms-iceberg-delegation-needs-e2e`）。 +- 本轮 workflow journal：`wf_8b907b93-e9f`（扩展范围）、`wf_9250330b-e81`（关闭接线生命周期）；前轮 `wf_72d1e505-75c`/`wf_62cc379e-c6e`/`wf_7e537094-44f`。 +- e2e 一律留连接器进 `SPI_READY_TYPES` 切换阶段统一补；异构网关 e2e 随 STEP 2/4（对齐 `hms-iceberg-delegation-needs-e2e`）。 diff --git a/plan-doc/per-statement-table-owner-port/designs/P1-implementation-design.md b/plan-doc/per-statement-table-owner-port/designs/P1-implementation-design.md new file mode 100644 index 00000000000000..8cc915afe6b99e --- /dev/null +++ b/plan-doc/per-statement-table-owner-port/designs/P1-implementation-design.md @@ -0,0 +1,180 @@ +# P1 实现设计（seam-by-seam）—— 引擎自持每语句 ConnectorMetadata 实例 + +> **范围 = 仅 P1 键石**：让一条语句、一个 catalog(一个属主连接器)在读/扫描/DDL/MVCC 路径上**恰好复用一个 memoized `ConnectorMetadata` 实例**,`connector.getMetadata` 保持纯工厂、唯一 memo 在 fe-core 管道,语句末**确定性 close**。写路径(7 缝)与工作集下沉、按身份分片缓存分别留 P3/P2/P4。 +> **本文仍不动代码**:是"可动工蓝图 + commit 切分"。目标架构全景见 [`trino-parity-metadata-redesign-design.md`](./trino-parity-metadata-redesign-design.md)。 +> 全部 `file:method:line` 来自本轮只读 grounding(workflow `wf_7e537094-44f`,已核实)。 + +--- + +## 0. P1 成功判据 + +- 一条查询里,对同一 `(catalogId, 属主连接器)`,`connector.getMetadata(session)` 只被真正调用**一次**;读/扫描/DDL/MVCC 的所有 seam 复用该实例。 +- 该实例在**查询结束、BE/pump 静默之后**被确定性 `close()`(P1 里 close 仍是 no-op,只验证接线正确);预编译 EXECUTE 重执行不泄漏;异构 HMS 网关下每 sibling 一实例。 +- **P1 字节中性**:`NONE`(离线/测试/无 ConnectContext)下逐次工厂=今日行为;perf delta≈0(语句内去重本就有,收益=单漏斗 + 无 per-connector memo + 确定性生命周期)。 +- 守门:见 §7。 + +--- + +## 1. 新增 / 改动的 SPI 与 fe-core plumbing（精确签名） + +### 1.1 `ConnectorStatementScope`（fe-connector-api,加一个默认方法) +现状:` T computeIfAbsent(String key, Supplier loader)` + `NONE`(`connector/api/ConnectorStatementScope.java:45,48`)。 +**加**(默认方法,`NONE` 天然逐次不缓存): +``` +// 便利入口:类型化 metadata memo；默认经 computeIfAbsent,故 NONE 下逐次跑 factory +default ConnectorMetadata getOrCreateMetadata(String key, Supplier factory) { + return computeIfAbsent(key, factory); +} +// 语句末确定性关闭；默认 no-op,故 NONE 恒惰性 +default void closeAll() { } +``` +> 值仍以 `Object` 存(fe-core 不认连接器类型,守 connector-agnostic 铁律)。 + +### 1.2 `ConnectorStatementScopeImpl`（fe-core) +现状:`ConcurrentHashMap` + `computeIfAbsent`(`connector/ConnectorStatementScopeImpl.java:38-44`)。 +**加** `closeAll()`:遍历 values,对 `Closeable`/`AutoCloseable` 者 best-effort 关闭(log-and-continue,仿 `QueryFinishCallbackRegistry` 隔离);**幂等**(关一次后清标记/清表,重复调用无害——因 close 会从多处触发,见 §4)。 + +### 1.3 静态 funnel（fe-core,新类 `PluginDrivenMetadata`) +``` +public static ConnectorMetadata get(ConnectorSession session, Connector connector) { + ConnectorStatementScope scope = session.getStatementScope(); // 默认 NONE + String key = "metadata:" + session.getCatalogId(); // 普通连接器 + return scope.getOrCreateMetadata(key, () -> connector.getMetadata(session)); +} +``` +- `NONE`(无 ConnectContext/StatementContext)→ `getOrCreateMetadata` 走 `computeIfAbsent` 的 NONE 实现 → **逐次 `connector.getMetadata(session)`,零跨语句缓存**(off-thread loader 即便误路由也安全)。 +- `session` 仍要照常构造(后续 `metadata.xxx(session, …)` 要它);**只 memoize `getMetadata` 的结果**。 +- 证据:`session.getCatalogId()` 恒有(`buildConnectorSession` 一律 `withCatalogId(getId())`,`PluginDrivenExternalCatalog.java:1118`);`ConnectorSession.getCatalogId():69`、`getQueryId():31`。 + +### 1.4 `ConnectorMetadata.close()`（已存在,无需新增) +`ConnectorMetadata extends 6 Ops + Closeable`,默认 `close() throws IOException` 为 no-op;**无连接器 override、无 fe-core 调用点**(`connector/api/ConnectorMetadata.java:248`)→ 开始 memoize + 在 closeAll 里调它是**生命周期安全**的。P1 不改各连接器 close(仍 no-op),只验证接线。 + +### 1.5 close 回调注册(见 §4;不新开 `StatementContext.close` 步) + +--- + +## 2. 66 处 seam 的改道规则 + +| 组 | 位置 | 数量 | P1 处置 | 说明 | +|---|---|---|---|---| +| **A 读** | `PluginDrivenExternalTable`(13)+`ExternalDatabase.getLocation`(1) | 14 | **改道 funnel** | 均 on-thread;connector=`getConnector()`,session=`buildConnectorSession()` | +| **A/排除** | `PluginDrivenExternalTable.getChunkSizes:1087` | 1 | **保持直调(fallback)** | 统计分析线程,常无 ConnectContext | +| **B DDL** | `PluginDrivenExternalCatalog` DDL(19)+`tableExist`(1)+name-mapping(2) | 22 | **改道 funnel** | on-thread;connector=字段 | +| **B/排除** | `listDatabaseNames:295`、`listTableNamesFromRemote:311` | 2 | **保持直调** | name-cache loader,off-ConnectContext | +| **C 扫描** | `PluginDrivenScanNode`(create:205 + 8 处 per-method) | 9 | **改道 + 存字段** | 见 §3 | +| **D 写** | PhysicalPlanTranslator×2 / InsertExecutor / BindSink×2 / IcebergMergeSink / IcebergRowLevelDmlTransform | 7 | **P1 不动,留 P3** | 见 §6(read-vs-write 复用) | +| **E MVCC** | `PluginDrivenMvccExternalTable`(143/358/725) | 3 | **改道 funnel**(带 null-ctx fallback) | 主 on-thread,少数后台 metadata-table 扫描 | +| **F 排除** | `initSchema:460`、`getColumnStatistic:1052`、`fetchRowCount:1153` | 3 | **保持直调** | 跨语句缓存 loader,off-ConnectContext | +| **misc** | ShowPartitions/ConnectorExecuteAction/CallExecuteStmt/JdbcQueryTVF | 4 | **改道 funnel** | on-thread 命令/TVF | +| **misc/排除** | `MetadataGenerator.dealPluginDrivenCatalog:1329` | 1 | **保持直调** | BE 驱动 fetch,可能无 ConnectContext | + +- **改道机械动作**:`ConnectorMetadata m = connector.getMetadata(session);` → `ConnectorMetadata m = PluginDrivenMetadata.get(session, connector);`(connector/session 每处都在作用域)。 +- **排除项本质安全**:funnel 在 `NONE`/null-ctx 下本就 fallback 直调;但排除项**保留显式直调**更清晰、且避免误把跨语句 loader 的结果塞进某个残留 scope。 +- **假阳性**:`datasource/test/TestExternalCatalog.java` 的 8 处 `catalogProvider.getMetadata()` 是测试用静态 Map,非 SPI,勿动。 +- **negative**:`planner/PluginDrivenTableSink.java` 无 `getMetadata` 调用(sink 从 translator 收 session+handle),D 组工作全在 PhysicalPlanTranslator。 + +--- + +## 3. 扫描节点（组 C）：存字段去 per-method 重取 + +`PluginDrivenScanNode` 已把 `connector`(:147)、`connectorSession`(:148)、`currentHandle`(:151) 存字段(构造期赋值 190-192,create 建 203-205)。 +- **加**懒字段:`private ConnectorMetadata cachedMetadata;` + `private ConnectorMetadata metadata(){ if(cachedMetadata==null) cachedMetadata = PluginDrivenMetadata.get(connectorSession, connector); return cachedMetadata; }`。 +- 8 处 per-method 重取(`convertPredicate:805`/`tryPushDownLimit:845`/`tryPushDownProjection:864`/`pinMvccSnapshot:909`/`pinRewriteFileScope:1054`/`pinTopnLazyMaterialize:1073`/`buildColumnHandles:1889`/`buildRemainingFilter:1926`)一律改调 `metadata()`。 +- **懒**(而非 create 传入):`JdbcQueryTableValueFunction.getScanNode` 直构造节点、手里无 metadata。 +- 8 处都跑在**单规划线程**(在并发 `appendBatch` 之前),但镜像现有 `volatile resolvedScanProvider` 模式加 `volatile` 更稳(off-path 若调 `metadata()` 防竞态)。 + +--- + +## 4. 确定性 close 接线（P1 最关键、含红队盲区①②的确切修法） + +> **⚠ 已更正（2026-07-19，workflow `wf_9250330b-e81` 取证 + 已实现于 commit `12f3e95239b`）**：本节原方案"注册点 = scope 首次创建时（`getOrCreateConnectorStatementScope`）"**会泄漏**——`captureStatementScope` 在**每次 on-thread 会话构建**就急切建 scope，而 DDL/`SHOW`/`DESCRIBE`/`EXPLAIN`/前台 `ANALYZE` 走 `Command.run` 建了 scope 却**永不到 `unregisterQuery`**，回调（连带 scope）永留无 TTL 的注册表。已改为**两层关闭**： +> - **主关闭**：注册点 = `PluginDrivenScanNode.getSplits`（挨着现有 read-txn 回调、同 queryId 键、对象捕获 `scope::closeAll`、跳过 `NONE`）。getSplits 只对**走协调器**的扫描/写/游标取数/内部查询触发，全部到 `unregisterQuery` → 无注册表泄漏、pump 静默后触发。 +> - **兜底关闭**：`StatementContext.close()` 加 `isReturnResultFromLocal` 守卫的 closeAll，覆盖不走协调器的 `Command` 语句（直连经 `ConnectProcessor.executeQuery` finally 已调 close；转发主节点在 `proxyExecute` 补 finally 调 close）。arrow-flight（异步返回）经守卫延后到其自身 query-finish close，不被提前关。 +> - **重置**：`resetConnectorStatementScope()` 改"先 closeAll 再置空"；`handleQueryWithRetry` 每次重试前重置 → 预编译 EXECUTE / 重试复用的 StatementContext 不会 memoize 进已关闭 scope。 +> 幂等（closeAll close-once）保证主+兜底双触发时恰好关一次。下面 §4.1/§4.2 保留原始分析（含为何不用 `StatementContext.close` 作**唯一**钩子的红队盲区①），但**注册点以本更正为准**。残留风险见 expanded-scope 文档。 + +### 4.1 钩子 = 现有 query-finish 回调,**不是** `StatementContext.close()` +- 注册:`QeProcessorImpl.registerQueryFinishCallback(String queryId, Runnable)`(`QeProcessorImpl.java:216`)。 +- 触发:`unregisterQuery` → `QueryFinishCallbackRegistry.runAndClear(DebugUtil.printId(queryId))`(`:212`),在 `StmtExecutor.finalizeQuery`(`:1034`,`updateProfile(true)` 之后)里调;而 `coordBase.close()`(`handleQueryStmt:1582`)已先驱动 `Coordinator.close:817`→`FileQueryScanNode.stop:695`→`splitAssignment.stop()` **栅栏住 off-thread pump**。⇒ 回调**严格在 BE drain + pump 栅栏 + profile 等待之后**触发,每 queryId 一次。 +- key 对齐:`connectorSession.getQueryId()` = `DebugUtil.printId(ctx.queryId())`(`ConnectorSessionBuilder.from:75`),与 `runAndClear` 同 key。 +- 先例:hive 读事务释放 `buildReadTransactionReleaseCallback`(`PluginDrivenScanNode.java:687`,在 `getSplits:1207` 无条件注册)——`scope.closeAll()` **完全镜像它**(含 `onPluginClassLoader` TCCL pin,若 closeAll 触及连接器加载对象)。 +- 为何 `StatementContext.close()` 错(盲区①):它只 `releasePlannerResources()`(`StatementContext.java:979`,故意不碰 scope,注释 :205);它从 `ConnectProcessor.executeQuery:410` 在 execute() 之后跑、无对 BE/pump 的栅栏;SQL-cache 路径 `parseFromSqlCache:469` 只 `releasePlannerResources` **不 close**;**EXECUTE 复用一个 StatementContext 跨多次执行、只在最后 close** → 挂 close() 会漏掉每次中间执行的 scope。 + +### 4.2 注册点 = **scope 首次创建时**(修红队未点透的覆盖漏洞) +- grounding 指出:读事务回调注册**只在 `getSplits`**,而 `getSplits` **只有扫描语句到达**;写/纯元数据/DDL/EXPLAIN 语句在 `captureStatementScope`(`:202`)创建了 scope 却从不扫描 → 若照抄"getSplits 注册",这些语句的 `closeAll` **永不注册**。 +- **修**:在 `StatementContext.getOrCreateConnectorStatementScope()`(`:656`,scope 懒建的唯一入口)里,首次建 scope 时**向 `QeProcessorImpl` 注册一次** `queryId → scope.closeAll()`。⇒ 任何触连接器的语句都恰好注册一个 closeAll。 +- **幂等**:多张表/多 session 共享一个 scope、重试(`handleQueryWithRetry` 换 queryId 重跑 `getSplits`)都可能多次触发 → `closeAll()` 必须 **close-once**。 + +### 4.3 EXECUTE 复用(盲区③) +- `ExecuteCommand.run`(`:95`)每次执行调 `resetConnectorStatementScope()`(`StatementContext.java:669`,当前**只置空不 close**)。**修:改成先 `scope.closeAll()` 再置空**——作为兜底(某次执行的 scope 若在 analysis 阶段建、但计划短路没走到注册/触发)。因主 query-finish 回调可能已关同一实例,再次印证 closeAll 必须幂等。 + +### 4.4 取消/超时(盲区④) +- 反 pump 由 `Coordinator.close()`(取消/超时走 `:1396`)→`scanNode.stop()`→`splitAssignment.stop()` 栅栏;`SplitSourceManager` 另有 GC-based WeakReference reaper 兜底(`:72`)。close 回调走 query-finish、在 coord.close 之后,故**已在 pump 栅栏之后**,无需额外机制;仅需保证异常路径也走到 `unregisterQuery`(现有 finally 已覆盖)。 + +--- + +## 5. HMS 异构网关 sibling 扇出（组内特例,gateway 现已 LIVE） + +> **事实纠偏**:hms 已在 `SPI_READY_TYPES`(commit `83585fd5097`,2026-07-17,`CatalogFactory.java:56`),网关**服务真实查询**;代码里 ~20 处 "dormant until hms enters SPI_READY_TYPES" 注释是**过期**的。 + +- 网关行为:`HiveConnectorMetadata.getTableHandle:344` 按格式探测,ICEBERG/HUDI 表**转发 sibling**、返回 sibling 原生 handle;之后每个方法按 `instanceof HiveTableHandle` 判别,非 hive handle 经 `siblingMetadata(session,handle)=siblingOwnerResolver.apply(handle).getMetadata(session)`(`:292`)转发,**~43 处 per-handle 点每次新建 sibling metadata**。 +- owner 解析:`HiveConnector.resolveSiblingOwner(handle)`(`:190`)按 `ownsHandle` 三路派发**已建的** volatile sibling 字段(iceberg 先于 hudi),孤儿 handle fail-loud。每网关**恰好一个** iceberg + 一个 hudi sibling(`getOrCreateIcebergSibling:488` DCL)→ sibling Connector **对象身份稳定**。 +- **坑**:三连接器**共享一个 `catalogId`**(`createSiblingConnector` 传 `this` context,`DefaultConnectorContext.java:174`)→ **catalogId 单独做 key 会把三连接器塌成一个 metadata、派错**。 +- **修**: + 1. **funnel key 加属主 label**:`"metadata:" + catalogId + ":" + owner`,`owner ∈ {hive,iceberg,hudi}`,**在 `resolveSiblingOwner` 返回后按命中臂取 label**(非预解析、非 identityHashCode)。 + 2. **sibling getMetadata 也进 funnel**:`HiveConnectorMetadata.siblingMetadata`/`icebergSiblingMetadata`/`hudiSiblingMetadata` 里,先解析属主 Connector(照旧、可 fail-loud),再 `scope.getOrCreateMetadata(key(catalogId, ownerLabel), () -> owner.getMetadata(session))` → getTableHandle 的 by-TYPE 转发与后续 per-handle 转发**共享一个** sibling metadata/语句。 + 3. **只存/返回 `ConnectorMetadata` 接口**,绝不 cast 具体类型(跨 loader CCE)。 + 4. 补异构网关 e2e(`hms-iceberg-delegation-needs-e2e`,e2e 在 `external_table_p2/refactor_catalog_param`)。 +- 说明:scan/write/procedure provider 是 Connector 级(不在 getMetadata funnel),scan provider 已 per-handle memoize(PERF-11/C14);它们只共用 `resolveSiblingOwner` 的身份解析。 +- **注意**:sibling funnel 调用在**连接器内部**(fe-connector-hive),fe-core 的 arch 门禁看不见 → 门禁只管 fe-core 两文件(§8);sibling 一实例由连接器侧单测守(§7)。 + +--- + +## 6. read-vs-write 复用决策：P1 只做读侧,写侧留 P3 + +- 风险:部分连接器期望**每事务 metadata**(trino-connector 已用 `getMetadata(session, txn)` 2 参重载,`TrinoConnectorDorisMetadata.java:104`);把一个 metadata 实例跨读(扫描)+写(sink/insert)共享,可能违反事务语义。 +- **P1 决策:组 D(7 处写 seam)不改道,留 P3**。理由: + - P1 里 metadata 仍是**无状态外壳**(工作集 P2 才下沉),故"读侧共享一个实例"对写侧无影响;写侧维持现状(`PluginDrivenInsertExecutor` 已把 `writeOps` 按写语句自缓存,`ensureConnectorSetup:206`)。 + - 把"读+写同一实例/事务"的语义留给 P3 与事务折叠**一起**设计(那时才把 `ConnectorTransaction` 并入实例),避免 P1 就踩事务语义。 +- ⇒ **P1 的"一语句一实例"覆盖读/扫描/DDL/MVCC;写臂在 P3 汇入。** 这是干净的增量。 + +--- + +## 7. 守门与测试（全有现成模板） + +- **加载计数守门(连接器侧,iceberg)**:仿 `IcebergStatementScopeTest`(`:51-103`)——loader 内 `AtomicInteger`,共享=1、`NONE`=2;远端 loadTable 计数用 `RecordingIcebergCatalogOps.log`。连接器侧**须 copy `TestStatementScope`**(不能 import fe-core,bash 门禁会挂)。 +- **跨 catalog / 跨 queryId 隔离**:同文件两个 `ScopeSession` 仅 catalogId / queryId 不同,断言非同实例(已有 `differentCatalogIdIsolatesTheLoad`/`differentQueryIdIsolatesTheLoad`)。 +- **fe-core 侧 funnel + 预编译无泄漏**:仿 `ConnectorStatementScopeTest`(`:35-94`),真 `StatementContext` + `ConnectorStatementScopeImpl`;`resetConnectorStatementScope` 后断言换实例且旧值不存;**加断言 closeAll 被调**(用可数 close 的假值)。 +- **HMS sibling 一实例 + 扇出不重载**:仿 `HiveConnectorSiblingTest`(`:67-107`,`buildCount==1`/close-once/fail-loud-not-memoized);驱动 getTableHandle→scan/write 转发,断言 sibling 只 loadTable 一次(`RecordingIcebergCatalogOps.log` size)。 +- **close-once / 游标取数 / 转发 / 重试**:P1 净新。可数 close 的 scope 值,经批式 pump(`PluginDrivenScanNodeBatchModeTest` 风格)驱动,断言早终止/异常路径 close 恰一次。 +- **NONE 离线**:`ConnectorSessionImpl` ctor 默认 NONE、`captureStatementScope` 无 ctx 返回 NONE → 离线测试自动落 NONE。 + +--- + +## 8. arch 门禁（enforce invariant 2,非仅约定） + +- 机制 = **bash grep 门禁**(仿 `tools/check-connector-imports.sh`,同风格 + 自测),**非 ArchUnit**(classpath 无,勿引)。 +- 规则:grep fe-core 的 `connector.getMetadata(` 调用形,**只允许** `PluginDrivenExternalCatalog.java` 与 `PluginDrivenExternalTable.java`(fe-core 里唯二合法 funnel 载体;funnel 自身 `PluginDrivenMetadata` 内部那一处 factory lambda 也在允许名单)。 +- 排除项(§2 的 7 处直调 loader)要么走 funnel 的 null-ctx fallback、要么在门禁里显式白名单(建议白名单,保留显式直调语义)。 +- 落点:`fe/fe-connector/pom.xml` 现有 exec 只扫 fe-connector 根;fe-core 域的新规则需**自己的 exec execution**,或用 checkstyle `Regexp`(checkstyle.xml 已用 Regexp 模块)限定 `org.apache.doris.datasource.plugin`。 +- 正则须锚定调用形,别误伤 API 定义处与 `getMetadataTableRows`。 + +--- + +## 9. 未决 / 需你确认点 + +1. **写侧留 P3**(§6):P1 不改 7 处写 seam,读侧先"一语句一实例"。认可否?(替代:P1 也把写 seam 改道,但要先答 read-vs-write 事务语义——我不建议在 P1 冒这个险。) +2. **注册点 = scope 创建时**(§4.2):在 `getOrCreateConnectorStatementScope` 里向 query-finish 注册 closeAll(覆盖写/DDL/EXPLAIN)。认可否?(替代:只在 getSplits 注册=会漏非扫描语句。) +3. **排除项处置**(§2/§8):7 处 off-ctx loader **保留显式直调 + 门禁白名单**,而非依赖 funnel 的 null fallback。认可否? +4. **P4 残留泄漏威胁模型**(总设计 §8.3⑤):属独立安全签字,不阻塞 P1。确认 P1 期间**不动** `SchemaCacheValue`/`latestSnapshotCache` 等缓存门禁。 + +--- + +## 10. P1 落地顺序（建议 commit 切分） + +1. **C1 地基**:`ConnectorStatementScope.getOrCreateMetadata/closeAll` 默认方法 + `ConnectorStatementScopeImpl.closeAll`(幂等)+ `PluginDrivenMetadata` funnel。含 fe-core 单测(memo/NONE/隔离)。 +2. **C2 close 接线**:`getOrCreateConnectorStatementScope` 注册 query-finish closeAll + `resetConnectorStatementScope` closeAll-before-null。含 close-once / EXECUTE / 取消 用例。 +3. **C3 读侧改道**:组 A/B/E/misc(排除 §2 标注项)+ 扫描节点组 C 存字段。含加载计数守门。 +4. **C4 HMS sibling**:key 加属主 label + sibling getMetadata 进 funnel。含 sibling 一实例守门 + 异构网关 e2e。 +5. **C5 arch 门禁**:bash/checkstyle 规则 + 自测。 +- 每个 commit 独立可编译、可回退;C1 后即"每语句一实例"(读侧),C4 后覆盖异构网关。**纯连接器无关地基 + fe-core 改道**,用户已豁免铁律 A。 diff --git a/plan-doc/per-statement-table-owner-port/designs/expanded-scope-phasing-and-security-decisions.md b/plan-doc/per-statement-table-owner-port/designs/expanded-scope-phasing-and-security-decisions.md new file mode 100644 index 00000000000000..5d7069553ce6de --- /dev/null +++ b/plan-doc/per-statement-table-owner-port/designs/expanded-scope-phasing-and-security-decisions.md @@ -0,0 +1,96 @@ +# 扩展范围定稿：分期 + 后台读穿纠正 + 缓存隔离升级为安全修复 + +> 本文承接 `P1-implementation-design.md` §9 的四个待确认点。用户 2026-07-19（session 3）就"是否把写入共用 / 后台线程传递 / 缓存隔离一并纳入当前工作"做了拍板，并触发一轮取证 workflow（`wf_8b907b93-e9f`，18 agents，recon + 对抗验证 + 综合）。本文记录**用户决策、取证结论、定稿分期**，并更正 `P1-implementation-design.md` §9 点 3 与总设计里"后台传递"的隐含框架。 +> **本文仍不动代码**。 + +--- + +## 0. 用户决策（2026-07-19 session 3，动码前） + +`P1-implementation-design.md` §9 四点的实际裁决（部分偏离原推荐）： + +| 点 | 原推荐 | 用户裁决 | 影响 | +|---|---|---|---| +| ① 写侧留后续 | 只做读侧 | **读写一起做**（但经取证 → 排在读取键石之后、独立成步） | P3 提前进入范围，但**分期**、不与键石混提交 | +| ② close 注册点 | scope 创建时 | **认可**（scope 创建时） | 不变 | +| ③ 后台加载点 | 显式直调 + 白名单 | **要求"把统一句柄传到后台线程"** → 取证判定该方向**不安全**，纠正为"显式读穿 + 修隐患" | 见 §2 | +| ④ 缓存隔离 | 本阶段不动、留后续签字 | **现在就处理** + 确认 **list ≠ load** | 缓存隔离升级为**真实安全修复**，独立安全 track，见 §3 | + +**综合裁决**：**分期推进**（非一次性全做）。三块都做，但排序 + 独立提交 + 独立验证。 + +--- + +## 1. 取证结论 A —— 读写共用一个元数据实例：可行且忠实 Trino + +- **Trino 已核实**：`CatalogTransaction` 每（事务, catalog）memoize **恰好一个** `ConnectorMetadata`，读规划与写入**共用它**；写入经共享事务句柄绑定，提交 = `connector.commit(txnHandle)`。我们的目标形状与之**完全一致**。 +- **机制**：把统一入口存的"裸 metadata"升级成 `CatalogStatementTransaction` 共持体（持 memoized metadata + 首次写时懒建的 `ConnectorTransaction`）。现成范本 = `ConnectorRewriteDriver`（co-hold metadata/session + `beginTransaction` + commit/rollback）。 +- **写臂 8 处 getMetadata 改道进同一入口**（均已核实为无状态薄壳的只读用法 + 一次起事务）：`PhysicalPlanTranslator.visitPhysicalConnectorTableSink:660`(INSERT) / `buildPluginRowLevelDmlSink:612`(DELETE/MERGE)、`PluginDrivenInsertExecutor.ensureConnectorSetup:206`、`PluginDrivenExternalTable.resolveWriteTargetHandle:133`（藏在"读"文件里的写专用点）、`BindSink.checkConnectorStaticPartitions:673`/`checkConnectorWritePartitionNames:712`、`IcebergRowLevelDmlTransform.checkPluginMode:112`、`PhysicalIcebergMergeSink.buildInsertPartitionFieldsFromConnector:303`。 +- **两条必守的正确性闸门**（非性能）： + 1. **按连接器保留"起写刷新"**，不一刀切。iceberg 复用语句内共享表（新鲜度靠 `newTransaction()` refresh）；**hive 起写必须保留其在鉴权上下文里当场 `hmsClient.getTable`**（`HiveConnectorTransaction.beginWrite:199/208-211`）——负责 ACID 事务表拒绝 + 权威起始快照。误换成扫描缓存表 = 正确性/授权 bug。 + 2. **身份一致性闸门**：iceberg 接 REST、session=user 时 metadata 烤进 per-user 委派操作（`IcebergConnector.getMetadata:256` `newCatalogBackedOps`）。读/写会话在同一语句本是同一用户，但**须显式断言二者身份指纹相等**再共用，否则理论上"拿 A 的对象执行 B 的写"。 +- **不做的事**：不统一两个 `ConnectorSession`（保留 `setCurrentTransaction` 绑定，更大且无正确性必要）；不引入事务协调器（外部写仍每语句 autocommit，1 StatementContext ≈ 1 外部事务）。 +- **HMS 兄弟**：sibling 入口必须覆盖 **beginTransaction** 路由（`HiveConnectorMetadata:1854`），不仅 getMetadata；键含属主 label，写事务从与读同一个 funnel-memoized 兄弟实例上 mint。 +- **风险**：iceberg session=user 身份错配（首要正确性闸门）；hive 起写刷新丢失；兄弟 downcast；提交/收尾**不得**再调 getMetadata 建第二个实例；maxcompute/jdbc 写壳仅旁证，接入前须确认无状态 + 事务可从共享 metadata mint；scope-map 生命周期须在语句末确定性 close/rollback 未提交事务，且幂等。 + +## 2. 取证结论 B —— "后台线程传递"被证伪，纠正为"读穿 + 修隐患" + +**用户原要求"把统一元数据对象传到后台线程"，取证判定不安全，应反向做。** + +- **为何不安全**：那几个后台加载点填的是**跨语句共享缓存**（活得比语句久）；把每语句实例塞进去、语句末 close → 共享缓存里留**已关闭对象**。后台线程是**全进程复用固定池**（`ExternalMetaCacheMgr` scheduleExecutor），一个查询关掉的资源可能被**无关查询**在同一池线程上误用 → 崩溃/串数据。**明确拒绝 InheritableThreadLocal / 在 worker 上 set**。 +- **正确模型（两分支，也正是 Trino）**： + - **语句派生的异步**（扫描 pump、将来写侧异步）→ **提交时捕获**：复用请求线程上建好的 session（`PluginDrivenScanNode.connectorSession` final 字段，`ConnectorSessionImpl.statementScope` 是 final，脱线程可达）。规则：新异步一律把请求线程 session/scope 传进闭包，**绝不在池 worker 上 `buildConnectorSession()`**（那里 `ConnectContext.get()==null` → 落 NONE）。 + - **真正跨语句的后台缓存加载器**（7 处）→ **一律读穿**：每次临时新建、**显式声明 NONE**（`ConnectorSessionBuilder.withStatementScope(NONE)` 或新增 `buildCrossStatementSession()`），绝不绑定语句实例。7 处 = `PluginDrivenExternalCatalog.listDatabaseNames:295`/`listTableNamesFromRemote:311`、`PluginDrivenExternalTable.initSchema:460`/`getColumnStatistic:1052`/`getChunkSizes:1087`/`fetchRowCount:1153`、`MetadataGenerator.dealPluginDrivenCatalog:1329`。 +- **真实隐患（顺手修）**：`fetchRowCount` **并非**只跑在后台——`AnalysisManager.buildAnalysisJobInfo:415/417` 在 **ANALYZE 语句的执行线程**上直调 `table.fetchRowCount()`，该线程有活的 StatementContext → 会**错误绑定到 ANALYZE 语句的作用域**。今天无害（值是 `Optional`），但一旦实例挂可关闭资源即 use-after-close。→ **对 7 处一律强制 NONE**，把"读穿"从"碰巧落在哪个线程池"变成**强制契约**。 +- **Trino 对齐**：Trino 后台走跨事务缓存（`CachingHiveMetastore`）读穿，不复用事务 metadata；我们**匹配**，并**额外加**一道显式-NONE 守门（因我们的捕获是 thread-driven，Trino 不需要）。 + +## 3. 取证结论 C —— 缓存隔离：确认 list ≠ load，升级为真实安全修复 + +**用户确认 list ≠ load** → "能列举但无权加载"的用户会经缓存命中读到别人授权才见的表结构/快照/分区。**这是真实越权，非纵深防御。** + +- **影响面小**：只涉及 **iceberg 的投影缓存 + fe-core 表结构缓存**；hive/paimon/hudi 无 SUPPORTS_USER_SESSION、无按用户授权轴，**不动**。 +- **两轴切分（对齐 Trino 的切线）**：**只对"不含凭证、授权敏感"的投影按身份分片**（表结构/快照 id/分区规格）；**含凭证的原始表 + FileIO 一律不跨语句缓存**（凭证会过期；身份轴 ≠ 令牌过期轴）。 +- **SPI 原语**：新增连接器无关的 `ConnectorSession.getIdentityShardKey()`（默认取 `getUser()`，源自 Doris 主体非令牌字节 → fe-core 不解析凭证，守铁律）。off-thread loader 也读它，防指纹在请求线程与异步线程间漂移。 +- **iceberg**：把 shard key 放进当前**未门控**的三个投影缓存的 Key——`latestSnapshotCache`/`partitionCache`/`formatCache`（`IcebergConnector` cache 块 ~200-232），仅 `isUserSessionEnabled()` 时填充（否则常量 → 非 session=user 目录字节不变）。关掉残留泄漏（总设计 §8.3⑤：`beginQuerySnapshot` 命中跳过 per-user loadTable）。 +- **fe-core 表结构缓存**（唯一引擎侧泄漏，`SchemaCacheKey` 仅按 nameMapping）：**分层**——(a) 先加 `shouldBypassSchemaCache(SessionContext)`，session=user + 委派凭证下**绕缓存现读**（镜像已安全的库/表名缓存 bypass），最小正确修；(b) 后续若性能需要再把 identityShardKey 织入 Key（触及广泛调用的 `getSchema` 签名 + off-thread loader，更侵入）。 +- **原始表缓存保持 OFF**（`tableCache` 两条件 null 门维持）；语句内复用仍靠 `IcebergStatementScope.sharedTable`（及 P2 后的实例字段）。**P4 恢复的是投影 RPC 的跨语句复用，不恢复 scan 期 loadTable**（那是 P2）。 +- **HMS 异构网关是最尖的洞**：hive 前门（无 SUPPORTS_USER_SESSION → fe-core 名字缓存 bypass 不触发）委派给 session=user 的 iceberg 兄弟时会泄漏；shard key 与 Key 选择须按**有效属主连接器身份**、且经兄弟 getMetadata 路由传播。**必须与 P1 的"键含属主连接器"一并落 + 异构网关 e2e。** +- **防漂移门禁**：加 arch/checkstyle 规则——授权敏感的跨语句缓存的 Key 在 session=user 下**必须**含 `getIdentityShardKey()`，把"加了新缓存忘了分片"从静默泄漏变成构建失败。 +- **Trino 对齐**：切线一致（只缓存无凭证投影、FileIO 每请求现挂）；Trino 的身份分片是 opt-in（impersonation），默认靠缓存之上的 access-control 层，故 **Doris 的"session=user 恒分片"比 Trino 默认更严**（需签字确认这是有意选择）。放置差异：Trino 分在**工厂层**（per-user 缓存实例），我们放**Key 里**（贴合 Doris 扁平有界缓存 + 总设计措辞），同等隔离、靠防漂移门禁补工厂模型的免费防漂移。 +- **待签字/待答**：授权新鲜度（TTL 内远端撤权仍命中，Trino 同性质，可接受但须签字）；manifestCache（默认 OFF，开启则须分片或声明与 session=user 不兼容）。 + +--- + +## 4. 定稿分期（不砍任何一块，只排序 + 独立提交/验证） + +> **为何不一次性全做**：①硬依赖——写入无入口可改道，直到读取键石落地（写严格下游）；②"后台传递"被证伪，非独立第三块，是键石 close 接线的一条正确性约束；③缓存隔离是独立安全工程（未决威胁模型 + 缺 SPI 原语 + 与重构正交），捆进 ~64 处机械改道会给签字施压、且越权回归无法二分定位。约六十多处改道 + 写 + 授权缓存同落刚稳定的性能热路径 = **不可二分的回归面**。 + +| 步 | 内容 | 依赖 | 验证 | +|---|---|---|---| +| **STEP 1 · 读取键石**（原 P1，含后台纠正） | 统一入口 `PluginDrivenMetadata.get` + `getOrCreateMetadata`/`closeAll`（幂等）+ 读/扫描/DDL/MVCC 改道 + 扫描节点存字段 + close 走现有 query-finish、注册在 scope 创建点 + `resetConnectorStatementScope` 先 closeAll 再置空。**7 处后台加载器显式强制 NONE 读穿 + 修 `fetchRowCount` ANALYZE 隐患**。 | — | 加载计数=1(NONE=N)；跨 catalog/queryId 隔离；游标/转发/重试/SQL-cache close 恰一次 | +| **STEP 2 · HMS 兄弟扇出**（P1-design §5） | 键 =(catalogId, ownerLabel)；兄弟 getMetadata **及 beginTransaction** 经同一入口；异构网关 e2e | STEP 1 | 每 sibling 一实例、加载计数=1 | +| **STEP 3 · 写入共用**（P3，拆两小步） | **3a** 无状态写点改道进入口（translator×2 / executor / resolveWriteTargetHandle / BindSink×2 / IcebergMergeSink / RowLevelDml）；**3b** `ConnectorTransaction` 归属上移到 `CatalogStatementTransaction`，定 commit/rollback vs closeAll 顺序 | STEP 1+2 | 3a 门：读/写会话身份等价（iceberg session=user）；保留 hive 起写刷新；保留 tx↔session 绑定 | +| **STEP 4 · 缓存隔离**（P4，独立安全 track，可与 1–3 并行启动） | `getIdentityShardKey()` SPI → iceberg 三投影缓存 Key 分片 + fe-core 表结构缓存 bypass(先)/分片(后) + 防漂移门禁；随 STEP 2 的属主键一并覆盖异构网关 | 威胁模型签字 + STEP 2（属主键） | 越权 e2e：can-list-cannot-load 用户命中不泄漏；异构网关 e2e | + +**纠缠点（STEP 3 时再定，现在不决）**：iceberg"起写复用已解析表"目前读 `IcebergStatementScope.sharedTable`，而该 side-car 在 P2 计划里要删。到 STEP 3b 请用户定：先接旧 side-car、P2 再搬 vs 先做个 iceberg 最小 P2 前置。 + +--- + +## 5. 对既有文档的更正 + +- `P1-implementation-design.md` §9 点 3：从"显式直调 + 白名单"**更正为"显式强制 NONE 读穿 + 修 fetchRowCount ANALYZE 隐患"**。§2 排除项组（A/排除、B/排除、F 排除、misc/排除）处置随之从"保持直调"细化为"显式 NONE session"。 +- 总设计"后台传递/off-thread 构造期捕获"表述保留正确（捕获机制没错），但**"把实例传给后台线程"这一用户方向被证伪**——后台跨语句 loader 必须读穿，仅语句派生异步捕获。 +- `P1-implementation-design.md` §9 点 4 与总设计 §8.3⑤：缓存隔离**不再是"本阶段不动 + 远期签字"**，而是**已确认 list≠load 的真实安全修复**，作为 STEP 4 独立 track，威胁模型签字仍需，但不再推迟到"某个远期"。 + +--- + +## 6. STEP 1 实现进展（2026-07-19 session 3 续） + +- **C1 地基**（commit `5b7312f9d1f`）：`ConnectorStatementScope.getOrCreateMetadata/closeAll` 默认方法 + `ConnectorStatementScopeImpl.closeAll`(幂等 close-once) + 静态漏斗 `PluginDrivenMetadata.get`。fe-core 单测：memo-once / NONE-each-call / 跨目录隔离 / closeAll 关一次。**字节中性**（无生产路径接进漏斗）。 +- **C2 关闭接线**（commit `12f3e95239b`）：**两层关闭**（见 `P1-implementation-design.md` §4 更正块）。主关闭=`PluginDrivenScanNode.getSplits` 注册 `scope::closeAll`（对象捕获、跳 NONE、同 read-txn queryId 键）；兜底=`StatementContext.close()` 的 `isReturnResultFromLocal` 守卫 closeAll（直连 `executeQuery` finally + `proxyExecute` 新增 finally）；`resetConnectorStatementScope` 改 closeAll-before-null；`handleQueryWithRetry` 每重试重置。单测：reset-先关 / close-本地关 / close-异步延后。**P1 关闭仍 no-op，本 commit 行为中性**。 +- 取证 workflow：`wf_9250330b-e81`（scope 创建普查 / unregisterQuery 覆盖 / 边界路径，各带对抗复核）。 + +### 关闭接线的残留风险（carry-forward，多为既有/共担） +1. **取消/超时非硬栅栏**：`SplitAssignment.stop()` 只置 `isStopped` 标志、不 join `scheduleExecutor` 上的批读 future；慢的在途 `planScanForPartitionBatch` 可能在 closeAll 清表后再 `computeIfAbsent` 塞值（不崩，但那值不被关）。**既有、与 read-txn 回调共担**，非本改动引入。P1 no-op 关闭下无实害；**关闭做实事前须硬化**（join pump 或 closed-guard computeIfAbsent）。 +2. **arrow-flight 异常断连**：`FlightSqlConnectProcessor.close` 不跑 → 注册表条目留存（无 TTL）。既有、共担。 +3. **待确认**：走协调器的 arrow-flight/内部查询若碰外部目录却**从不走 getSplits**（纯 information_schema / 某些元数据 TVF）→ 无主关闭 + arrow-flight 跳兜底 → 可能泄漏。需在 STEP 3/后续确认这类路径必走 getSplits 或本地兜底。 +4. **🔴 TCCL 自钉扎（硬前置，给"关闭做实事"的那一步）**：本 commit 关闭是 no-op 故无需；一旦某连接器把 `ConnectorMetadata.close()`（或 P2 的 FileIO/Table）做成实事，**主关闭（getSplits 注册的回调）与兜底（StatementContext.close/proxyExecute）两处都必须把 TCCL 钉到 provider 的插件 classloader**（同 read-txn 释放回调 + `TcclPinningConnectorContext` 的 locus 纪律），否则跨类加载器 split-brain。**连接器 `close()` 自钉扎是首选**（fe-core 保持 connector-agnostic、不持 classloader）。 diff --git a/plan-doc/per-statement-table-owner-port/designs/trino-parity-metadata-redesign-design.md b/plan-doc/per-statement-table-owner-port/designs/trino-parity-metadata-redesign-design.md new file mode 100644 index 00000000000000..92e30add0200b4 --- /dev/null +++ b/plan-doc/per-statement-table-owner-port/designs/trino-parity-metadata-redesign-design.md @@ -0,0 +1,228 @@ +# 目标架构设计：把外部表元数据层重构成"每语句/每事务 metadata 实例"（对齐并超越 Trino） + +> **本文定位**：这是"是否把 Doris 表解析/元数据层重构成 Trino 式架构"专题讨论的**设计草案**。 +> **前提（用户 2026-07-19 拍板）**：本设计**不受任何现有铁律/约束限制**——可改任何模块（fe-core / planner / 连接器）。唯一目标：设计一个**至少和 Trino 一样合理、能更好则更好**的架构，不被当前实现绑住。 +> **本轮仍不动任何生产代码**：先出设计、用户确认方向后再实现。 +> 现状事实全部经一轮只读并行 recon + 对抗验证取证（workflow `wf_72d1e505-75c`），下引 `file:method:line` 均来自该轮核实。 + +--- + +## 0. 结论先行 + +1. **可行，且比上一轮结论文档判断的容易得多。** 上一轮 §4.3 的关键前提"Doris 的 `*ConnectorMetadata` 是每 catalog 长期共享单例"**是错的**（见 §1 更正）。真相是:`ConnectorMetadata` 现在是**每次调用新建、即用即弃的无状态外壳**,生命周期比"每语句"还短。要把它变成"每语句一个",不是要拆一个顽固单例,而是**给一个本就无根的临时对象钉一个语句级的家**——这恰恰是 Trino 的做法。 +2. **Doris 现状已经具备 Trino 四条不变量里的两条半**:句柄不可变且携带事实(不变量③,已满足);语句级宿主 `StatementContext` 已存在且已托管 MVCC pin 与连接器作用域(不变量①的宿主已就位);缺的是"引擎在管道层 memoize 唯一 metadata 实例"(不变量②)和"生命周期即事务、按用户天然隔离"(不变量④)。 +3. **目标架构 = 引擎自持的"每语句(每事务)metadata 实例" + 其下一层"跨语句、按身份分片的共享缓存"**。前者给结构化的"一语句一次加载"(比 Trino 的 iceberg 更硬),后者给按用户的跨语句复用(补上今天 session=user 下缓存被全关的窟窿)。 +4. **诚实的收益定位**(必须带着走的老结论):"每语句实例"本身**不解决**按用户的**跨语句**性能——那要靠"按身份分片的缓存"(分期里的 Phase 4)。每语句实例真正的价值是**架构自洽 + 结构性正确**(把今天散落的 6 处"这个值是不是跨用户敏感"的门禁,收敛成一条结构性质:实例是每语句、天然单用户)。 + +--- + +## 1. 现状的真实形状（经核实，含一处对上一轮文档的更正） + +### 1.1 三层生命周期（这是理解一切的钥匙） + +| 对象 | 生命周期 | 持有什么 | 对应 Trino | +|---|---|---|---| +| **`Connector`** | **每 catalog 一个长期单例**(`transient volatile` 字段,仅 ALTER/重启/close 重建) | **全部状态**:各类跨查询缓存(iceberg 的 snapshot/table/partition/format/comment/manifest;paimon 的 snapshot+schemaAt;hive 的 fileListing;maxcompute 的 partition)、`ConnectorContext`、鉴权 | ≈ Trino 的 `Connector`(每 catalog 单例) | +| **`ConnectorMetadata`** | **每次 `getMetadata(session)` 调用新建、即弃** | **几乎无状态**,是把连接器单例的缓存注进来的**薄委派外壳** | ≈ Trino 的 `ConnectorMetadata`——但 Trino 是**每事务一个并被引擎 memoize**,Doris 是**每次调用一个、从不 memoize** | +| **`ConnectorSession`** | **每次 `buildConnectorSession()` 新建**(一条语句 ~26–63 次),且**贵**(每次 `VariableMgr.toMap()` 全量反射 dump + 全局读锁) | queryId、委派凭证、捕获的语句作用域引用 | ≈ Trino 的 `ConnectorSession`——但 Trino 的是**廉价不可变请求上下文**,Doris 的是**昂贵且被反复重建** | + +证据:`PluginDrivenExternalCatalog.java:102`(connector 字段)、`IcebergConnector.getMetadata:256`(`return new IcebergConnectorMetadata(...)` 每次新建,7 个连接器皆然,grep 无任何 `ConnectorMetadata` 类型字段)、`ConnectorSessionBuilder.java:158-179` + `VariableMgr.toMap:940-959`(每次 build 的反射开销)。 + +> **⚠ 对上一轮结论文档的更正**:`designs/recon-findings-and-trino-refactor-groundwork.md` §4.3 与 `HANDOFF.md` 写的"Doris 连接器/`*ConnectorMetadata` 是长期共享单例(每 catalog 一个、从不按语句造)"——**把 `Connector`(确是单例)和 `ConnectorMetadata`(其实是每调用即弃)混为一谈了**。对抗验证判定该表述 **REFUTED**(证据:`IcebergConnector.getMetadata:256` 等 7 处)。这个更正是**利好**:让 metadata 变成每语句,不需要打破单例,只需给一个"生命周期本就是自由变量"的临时对象钉个语句级的家。 + +### 1.2 规划器怎么拿元数据(每处都在重造) + +从 SQL 到执行,每个 seam 都在重复同一个三元组:**长期 catalog 单例 → 长期 Connector 单例 → 新建 `ConnectorSession` → `connector.getMetadata(session)`(每调用新 metadata)→ 从 metadata 拿一个不可变 `ConnectorTableHandle`**。**没有任何"每语句 metadata 对象"被引擎穿针引线地传下去。** + +需要改道的 seam(recon 已枚举,择要): +- **读**:`PluginDrivenExternalTable` 里 ~17 处(`initSchema`/`getColumnStatistic`/`fetchRowCount`/`getNameToPartitionItems`/`toThrift`/`getComment`/`listPartitions`…),每处都 `buildConnectorSession`+`getMetadata`。 +- **扫描**:`PluginDrivenScanNode` 在 `create()` 把 session/connector/handle 捕获成**字段**(单扫描节点内共享),但之后每个方法(`applyFilter`/`applySnapshot`/`planScan`/`getColumnHandles`…)仍 `connector.getMetadata(session)` 重取。 +- **写**:`PhysicalPlanTranslator.visitPhysicalConnectorTableSink`(自建 session)、`PluginDrivenTableSink.bindDataSink`(planWrite)、`PluginDrivenInsertExecutor`(**第二个** session + `writeOps` 字段 + `beginTransaction`)。写路径**两个独立 session**,仅靠把 `ConnectorTransaction` 绑到 sink 的 session 上勉强串起来。 +- **异步缓存 loader**(schema/rowCount/columnStats):**在没有 `ConnectContext` thread-local 的线程上跑**——任何"每语句对象"必须能不经 thread-local 够到(现有作用域用**构造期捕获**已解决此问题,可复用)。 + +### 1.3 已就位的"半个 Trino 形状"(利好) + +- **句柄已是 Trino 模型**:`ConnectorTableHandle` 不可变,由 `applyFilter/applyProjection/applyLimit/applySnapshot/applyRewriteFileScope/…` 返回**新句柄**逐步精化——这正是 Trino 的 `ConnectorTableHandle`-update 模式。**Trino 不变量③(事实随句柄前向流动、规划期不重解析)Doris 已满足。** +- **语句级宿主已存在**:`StatementContext` 每语句在 `StmtExecutor` 构造时新建、挂到 `ConnectContext`,已托管 MVCC 快照 pin(`StatementContext.snapshots`,一语句一次解析、读写共享)与连接器作用域,随语句 GC。`ConnectorStatementScope`(挂在它上的 `ConcurrentHashMap` memo 竞技场)+ iceberg 的 `IcebergStatementScope` 已用它做到"一语句一张表只加载一次"。 +- **但作用域只是"半成品"**:它是**无类型的 String→Object 竞技场**,不是**有类型的每语句 `ConnectorMetadata`**;**规划器不经它取表**(规划器走的是跨语句的 `ExternalCatalog` 元数据缓存);只有 iceberg 一个连接器手写键去消费它。也就是说"读规划的取表路径"和"每语句加载归属者"**今天还不是同一条缝**——而 Trino 把它们合成了一条。 + +### 1.4 事务模型(现状割裂) + +- **外部无统一事务管理器**:每个 `ExternalCatalog` 自持一个 `TransactionManager`;plugin 目录一律用 `PluginDrivenTransactionManager`,把 commit/rollback/close 委派给**连接器在 `beginTransaction` 才晚建**的 `ConnectorTransaction`(`session.allocateTransactionId()`),绑到 session、`onComplete` 提交/`onFail` 回滚。 +- **`ConnectorTransaction` 是糟糕的 span 宿主**:出生太晚(仅 beginWrite)、只在写臂、读/规划期根本看不见。 +- **内部 OLAP 是另一条路**:`GlobalTransactionMgr` + 每连接 `TransactionEntry`;`GlobalExternalTransactionInfoMgr` 只是个 id→Transaction 的注册表(供 BE→FE RPC 查),**不是协调器**。内外无共享抽象。 +- **外部写实际上是"每语句 autocommit"**:`ConnectorTransaction` 在一条语句内建、在该语句 `onComplete` 提交 → **一个 `StatementContext` ≈ 一个外部事务**。这让 `StatementContext` 成为 span 宿主是**自然且正确**的。 + +--- + +## 2. Trino 为什么优雅（四条不变量）+ 两处弱点 + +经源码核实(`trinodb/trino` master,下引类名/行号来自该轮 web recon): + +**四条不变量:** +1. **事务 = 元数据的身份与生命周期单位**。无显式 BEGIN 的语句跑在 autocommit 事务里;每个被触达的 catalog,`TransactionManager` 懒建并 memoize **一个** `CatalogMetadata`(`activeCatalogs` map,`CatalogHandle` 为键);其内 `CatalogTransaction` 用 `@GuardedBy(this)` 字段 memoize **恰好一个** `ConnectorMetadata`(`connector.getMetadata(session, txnHandle)` 只调一次)。commit/rollback → `connector.commit/rollback` → 事务移除。**无人工失效、无跨语句泄漏。** +2. **单一漏斗、单一实例**。所有元数据调用走 `Session → MetadataManager → TransactionManager → CatalogMetadata → CatalogTransaction`。**memoize 发生在管道层,不在连接器里** → 一语句一实例由**引擎**保证,与连接器自觉性无关。 +3. **事实随句柄前向流动、不靠重载**。`getTableHandle` 一次解析,快照 id/schema/分区 spec 装进不可变 `ConnectorTableHandle`,穿过 analyze/optimize/execute 直到 `beginInsert/beginMerge`。规划期从不重解析。 +4. **身份是事务级的 → 按用户隔离白拿**。`ConnectorMetadata` 用 `session.getIdentity()` 构建;`ConnectorSecurityContext = txnHandle+identity+queryId`;事务是每 session 的,故 metadata **从不跨用户共享**——隔离是事务模型的副产品。 + +**两处弱点(Doris 应超越):** +- **弱点 A**:每事务 `ConnectorMetadata` 被 memoize 了,但**加载到的 Table 对象没有**(iceberg)。`IcebergMetadata` 只缓存统计;`getTableHandle/getTableMetadata/getTableProperties` 各自重调 `catalog.loadTable()`,把去重下推到**每事务 `TrinoCatalog` 上有界可淘汰的** `tableMetadataCache`(maxSize ~1000)。**多表语句会淘汰重载 → "一语句一次加载"是软的、非结构性的**(直接坐实了"一条语句 loadTable 3–4 次靠下层缓存兜")。 +- **弱点 B**:core `CatalogTransaction` 已保证 `getMetadata` 只调一次,Hive 连接器却**又在 `HiveTransactionManager.MemoizedMetadata` 里重复实现一遍**每事务 memo。冗余耦合。 + +--- + +## 3. 目标架构（Doris 版：对齐 Trino 的不变量，并针对性超越其弱点） + +### 3.1 核心：引擎自持的"每语句/每事务 metadata 实例"，memoize 在管道层 + +把今天"每次调用新建即弃"的 `ConnectorMetadata`,改成**每(语句, catalog)一个、由引擎懒建并 memoize、语句/事务结束时确定性销毁**。 + +- **宿主 = `StatementContext`**(外部写=每语句 autocommit,一 `StatementContext` ≈ 一外部事务;宿主已在,已托管 MVCC pin 与作用域,已有正确 GC/reset)。 +- **管道层单一漏斗**:在语句 span 上加 `ConnectorMetadata getOrCreateMetadata(catalogId)`,懒建、memoize、语句内所有 seam 复用同一实例——**对应 Trino 不变量②,且 memo 在引擎侧,连接器零自觉性要求**(直接规避 Trino 弱点 B:连接器不再各自手写 memo)。 +- **off-thread 可达**:实例引用像作用域一样在 `ConnectorSession` **构造期捕获**,异步 scan pump / 缓存 loader 无 thread-local 也够得到。 +- **确定性销毁**:`StatementContext.close()` 逐 catalog `metadata.close()`(对齐 Trino 的 commit/rollback 即销毁;比今天纯靠 GC 更干净)。 + +因为 `ConnectorMetadata` 今天已是"无根的临时外壳"(designNotes 原话:"除了缓存,没有任何东西把它钉在每调用"),且句柄已满足不变量③,这一步在结构上**主要是引擎侧改道 + 把缓存归属从单例挪到实例**,SPI 方法签名基本不用动(方法本就吃 `(session, handle)`)。 + +### 3.2 超越点①：结构性的"一语句一次加载"(打败 Trino 弱点 A) + +把**每语句工作集**(一次加载的 raw Table、扫描→写的删除清单桥、列句柄)从"连接器单例上的缓存 / 无类型 String→Object 作用域",**上移成每语句 metadata 实例上的字段**。 + +- iceberg 的 `IcebergStatementScope.sharedTable`/`rewritableDeleteSupply` 从"外挂 side-car"变成"实例的字段" → **加载一次成为硬的结构性质**(实例在,表就在;不是有界可淘汰缓存)。**这比 Trino 的 iceberg 更硬。** +- paimon 的胖句柄 `PaimonTableHandle.paimonTable` 溶进实例字段,句柄回归纯坐标(iceberg 已做过同款)。 + +### 3.3 超越点②：两级缓存 —— 每语句工作集 之上/之下 各司其职 + +| 层 | 位置 | 作用 | 对应 Trino | +|---|---|---|---| +| **每语句工作集** | 每语句 metadata 实例的字段 | 语句内一次加载、读写共享、扫描→写桥;**天然单用户单语句** | Trino 每事务 metadata 的缓存(但 Doris 做成硬字段) | +| **跨语句共享缓存** | Connector 单例(或专门的 caching 层),**按身份分片** | 跨语句复用;**值敏感处把 identity 放进 key** | Trino 的 `CachingHiveMetastore` / `TrinoCatalog` 缓存 | + +关键洞察(对齐 Trino 不变量④,并补 Trino 未系统化之处): + +- **正确性坍缩成一条性质**:每语句实例**天然单用户**(一语句一用户),故其内部一切缓存**永不跨用户** → 今天散落的 ~6 处门禁(`SUPPORTS_USER_SESSION` 分支、`shouldBypassTableNameCache/DbNameCache` 两钩子、`IcebergTableCache` 置 null、`IcebergCommentCache` 门禁、`IcebergStatementScope` side-car)**收敛成"实例是每语句/单用户"这一条结构性质**,删掉一整类"新加了个缓存忘了 gate"的 bug。 +- **性能补窟窿在下层**:今天 session=user/凭证目录下,跨语句缓存被**整个关掉**(`IcebergTableCache=null` 等),每条语句都为每张表多付 1 次 `loadTable` + 1–2 次 list RPC。**把下层跨语句缓存改成按身份分片(identity 进 key)** → 按用户的跨语句复用安全恢复。**这才是按用户场景真正的性能收益所在。** + +### 3.4 超越点③：把写事务并进同一个 span + +- `beginWrite` 从"重新 `loadTable`"改成"**从每语句实例取已解析的表**"(保留 openTransaction 的 refresh 兜新鲜 OCC 基底)。 +- `ConnectorTransaction` 不再是"晚建 + 绑 session"的孤儿,而是**由每语句 metadata 实例/span 持有**;commit/rollback 由 span 在语句末驱动。写路径两个 session 的割裂被收编。对齐 Trino"metadata 即事务、写方法吃句柄、commit=connector.commit(txnHandle)"。 + +### 3.5 超越点④(远期/可选)：统一内外事务协调器 + +- Trino 只有外部连接器,没有"内部表"二元性;Doris 有 `GlobalTransactionMgr`(OLAP) 与 `PluginDrivenTransactionManager`(外部)两套。**把二者统一到一个事务协调面**,让外部写能参与多语句 `BEGIN..COMMIT`(span 挂 `ConnectContext`、`StatementContext` 持每语句视图,即"两级 span")——这是 Doris **特有**的自洽升级(不是"打败 Trino",是补 Doris 的割裂)。**最深、风险最高、对"达到 Trino 平价"非必需**,列为远期。 + +### 3.6 附带清理(与主线解耦，可先落) + +- **`ConnectorSession` 变廉价 + 不可变**:每语句 memoize 一次 var-map 快照,消灭 ~26 次 `VariableMgr.toMap` 反射 dump;把唯一可变字段(transaction 槽)移到 metadata/事务对象上,恢复 session 全不可变。**纯性能 + 简化,不依赖 metadata 生命周期改造,可最先落。** + +--- + +## 4. 可行性与风险 + +### 4.1 为什么可行(且比原判断容易) +- `ConnectorMetadata` 本就是每调用即弃的无根外壳 → **给它钉个语句级的家,不需打破任何单例**(最大的"想象中的拦路虎"不存在)。 +- 句柄已不可变、已 fact-carrying → **Trino 不变量③已满足**,不用重写。 +- `StatementContext` 已是被验证的 span 宿主(MVCC pin + 作用域已在其上),off-thread 可达已由"构造期捕获"解决 → **不变量①的宿主与最难的 off-thread 问题都已就位**。 +- seam 虽多但**窄而齐**:一切走 `getMetadata / getScanPlanProvider / getWritePlanProvider / beginTransaction`,fe-core 从不跨调用持有 metadata 引用(唯一例外是写执行器的 `writeOps` 字段)→ 改道是机械但可枚举的工作。 + +### 4.2 风险(诚实列出) +1. **回归面**:~40 处读热路 seam 改道,踩在刚稳定的读热缝上(PERF 系列)。→ 缓解:分期、每期加"加载计数守门"回归。 +2. **确定性销毁 vs off-thread**:实例不能在异步 scan pump 还在用时提前 close。→ 缓解:沿用作用域的"随语句 GC + 不在 close() 里清工作集"纪律,只在确证无 off-thread 引用处做确定性 close。 +3. **异步缓存 loader 无 thread-local**:schema/rowCount/stats loader 跑在别的线程、且填的是**跨语句**缓存。→ 这些本就该留在"下层跨语句缓存",每语句实例**读穿**到它们(对齐 Trino 每事务 metadata 读穿 `CachingHiveMetastore`),不强行纳入每语句实例。 +4. **Phase 4 的身份分片需威胁建模**:哪些值是用户敏感(凭证/可见性/授权)要 identity 进 key,哪些是纯元数据可共享(snapshot/format)。recon 已留两个 open 问题(session=user 下 `SchemaCacheValue` 仍开、latest-snapshot 仍跨用户共享——是否可接受需签字)。 +5. **一语句多事务?**(open Q):多目标 MERGE、或 HMS 异构网关委派兄弟连接器,是否一条语句会 mint 多个 `ConnectorTransaction`?若是,"一 StatementContext = 一事务"需放宽成"一实例/catalog"。→ 设计上按 (语句, catalog[, 目标表]) 建键即可容纳。 + +--- + +## 5. 分期（避免一次性大爆炸；每期独立可验证） + +| 阶段 | 内容 | 独立价值 | 打败 Trino? | 风险 | +|---|---|---|---|---| +| **P0 · 解耦的廉价前置** | `ConnectorSession` 每语句 memoize var-map(灭 ~26× 反射 dump);为后续把 transaction 槽移出 session 铺垫 | 纯性能 + 简化 | — | 低 | +| **P1 · 键石:引擎自持每语句 metadata 实例** | span 上加 `getOrCreateMetadata(catalogId)`,管道层 memoize,读/规划/DDL/list 全改道走它;确定性 close;off-thread 构造期捕获 | 一语句一实例(不变量①②) | 弱点 B(单漏斗、连接器零自觉) | 中(改道面广) | +| **P2 · 工作集上移 + 删 side-car** | 一次加载的 Table/扫描→写桥/列句柄 变成实例字段;删 `IcebergStatementScope` 的 String-key 用法;paimon 胖句柄溶进实例 | 结构性一次加载 | **弱点 A(硬 load-once)** | 中 | +| **P3 · 写事务并入 span** | `beginWrite` 取实例已解析表(非重载);`ConnectorTransaction` 由实例/span 持有;收编两 session 割裂 | 读写同一 span、写路径自洽 | 对齐 Trino metadata=事务 | 中-高 | +| **P4 · 按身份分片的跨语句缓存** | 下层缓存 identity 进 key;删 `SUPPORTS_USER_SESSION` 全关门禁,改成"缓存按用户分片";补 session=user 跨语句性能窟窿 | **按用户跨语句性能**(真正性能收益) | 超越 Trino 的 all-or-nothing gate | 中(需威胁建模) | +| **P5 · 统一内外事务协调器(远期)** | 一个事务面桥接 OLAP + 外部;外部写参与多语句 BEGIN..COMMIT;两级 span | 内外自洽、多语句外部事务 | Doris 特有自洽(非平价必需) | 高 | + +- **达到 Trino 平价 = P1–P3**;**超越 Trino = P2(硬 load-once)+ P4(按用户缓存)**;**P5 是远期战略,非平价必需**。 +- 建议落地顺序:P0(热身)→ P1(键石)→ P2 →(P4 与 P3 可并行/择序)→ P5 视产品需要。 + +--- + +## 6. 用户已拍板的点（2026-07-19） + +1. **终点 = 平价 + 超越(P1–P4)**;不含 P5(统一内外事务)。 +2. **首个落地单元 = 直上 P1 键石**(跳过 P0 热身)。 +3. **销毁 = 语句末确定性 close**(对齐 Trino,接受 off-thread 约束)——**但见 §8 盲区①:实际须挂在现有 query-finish 钩子,而非 `StatementContext.close`。** +4. **先跑一轮多架构师对抗红队再定稿**——已完成,结论见 §8。 + +--- + +## 7. 参考 + +- 本轮 recon+对抗验证 workflow:`wf_72d1e505-75c`(journal 在 `.../subagents/workflows/wf_72d1e505-75c/`),含 metadata 生命周期 / planner 取数路径 / 多租户缓存 / span+txn 模型 / Trino 精确机制 五组核实结论 + 4 条支撑事实的对抗判定。 +- 上一轮结论(部分被本轮 §1 更正):`recon-findings-and-trino-refactor-groundwork.md`。 +- 架构记忆:`iceberg-table-resolution-cache-scoping`(缓存作用域五纪律 + StatementContext=每语句 owner + Trino 协同)。 +- Trino 源:`InMemoryTransactionManager` / `CatalogTransaction` / `CatalogMetadata` / `MetadataManager` / `HiveMetadataFactory` / `IcebergMetadata`(master)。 + +--- + +## 8. 红队结论与定稿（多架构师对抗,workflow `wf_62cc379e-c6e`,2026-07-19） + +4 位架构师(span 中心 / 缓存中心 / 忠实 Trino / 最小面)各出一版目标架构,4 位评审(Trino 忠实度 / 迁移风险+off-thread / 多租户正确性 / 完整性)横向打分 + 找致命伤 + 挑最佳点。 + +### 8.1 排名 +| 评审轴 | 第1 | 第2 | 第3 | 第4 | +|---|---|---|---|---| +| Trino 忠实+超越 | **span (85)** | 忠实Trino (83) | 最小面 (77) | 缓存 (58) | +| 迁移+off-thread | **span (82)** | 最小面 (76) | 忠实Trino (66) | 缓存 (58) | +| 多租户正确性 | 缓存 (84) | **span (80)** | 最小面 (71) | 忠实Trino (64) | +| 完整性 | **span (64)** | 忠实Trino (61) | 最小面 (55) | 缓存 (45) | + +**胜出 = span 中心**(3/4 轴第一):复用**已验证 off-thread 可达**的现有 `ConnectorStatementScope` + `getStatementScope()` 到达路径;`connector.getMetadata` 保持纯工厂;**唯一 memo 在 fe-core 管道**(invariant 2 最纯);P1 近字节中性、可回退、风险最低。 + +> 一处被评审核实的关键点:现 `ConnectorSessionImpl` 捕获的是 **`ConnectorStatementScope` 接口**、不是 `StatementContext` 本身。故"经 getStatementScope 到达"是**被证明的**路径;而"把新管理器直接挂 StatementContext"(忠实 Trino 派原案)**不在**这条已证 off-thread 路径上,须额外补捕获——这也印证应以 span 派为骨架。 + +### 8.2 嫁接进定稿的各家最佳点 +1. **忠实 Trino 派的 `CatalogStatementTransaction`**:把"每(语句,catalog)metadata 实例"与"写事务"做成**同一个持有者**(invariant 1 最紧、P3 折叠最干净)。**弃用**其 `ConnectorSession.getMetadata(connector)` SPI(分层异味 + fallback 少测)。 +2. **缓存派的"元数据/凭证拆分"(P4 唯一正确的威胁模型)**:缓存**不含凭证的投影**(schema/snapshotId/分区 spec/授权名单)按 catalog 共享、**每请求现挂新 FileIO**;只对授权敏感投影按身份分片;**vended raw Table 一律不跨语句缓存**。身份轴 ≠ 令牌过期轴。 +3. **最小面派的**:`ConnectorSession.getIdentityShardKey()` 集中指纹(off-thread loader 复用防漂移)+ `ConnectorMetadata.close()` 默认 no-op + 扫描节点把 memoized metadata 存字段去掉 per-method 重取;`statementSession()` 消灭 ~26 次 `VariableMgr.toMap` 反射 dump(**列为独立性能项,不进 P1** 以保 P1 字节中性)。 +4. **arch/checkstyle 门禁**:禁止在 funnel 之外直接调 `connector.getMetadata`(把 invariant 2 从"约定"变"结构强制";须对 HMS sibling 布线显式放行/改道)。 + +### 8.3 五处四家都漏的公共盲区(定稿必修,①②为硬伤) +1. **🔴 确定性 close 挂错生命周期**。四家都想挂 `StatementContext.close()`/StmtExecutor finally。但每查询连接器资源释放本走 **`registerQueryFinishCallback`**(`unregisterQuery`→`finalizeQuery`,profile 等待之后=pump/BE 静默之后触发,现已用于提交 hive 读事务+放锁)。二者只在简单 autocommit 单语句下重合;**游标取数 / 转发 master / 重试 / SQL-cache 复用(只 releasePlannerResources 不 close)** 下会**过早/重复/永不**触发 → 关掉 off-thread pump 仍用的 Table/FileIO。**修:close 走现有 query-finish 钩子。** +2. **🔴 HMS 异构网关 sibling 扇出**。`HiveConnectorMetadata` 在 ~25 处 per-handle 点调 `siblingOwnerResolver.apply(handle).getMetadata(session)`,每次**在连接器内部新建 sibling metadata、funnel 看不见** → 网关一开:invariant 2 破、P2 一次加载破(sibling Table 重建 ~25 次)、catalogId 单键装不下三连接器、lint 门误报/漏保证。**修:memo 键改 `(catalogId, 属主连接器身份)`;sibling getMetadata 也路由进同一 funnel;补异构网关 e2e(记忆 `hms-iceberg-delegation-needs-e2e`)。** +3. **预编译 EXECUTE 复用泄漏**:`resetConnectorStatementScope()` 先置空不 close → P2/P3 后跨执行泄漏 FileIO/事务。**修:reset 先 closeAll 再置空。** +4. **取消/超时 reaper**:close 须栅栏在 `SplitSourceManager` 注销之后(reaper 可能 close 后才懒调 planScan),不能靠含糊"pump join"。 +5. **P4 残留泄漏(威胁模型待定)**:`latestSnapshotCache`/`partitionCache`/`formatCache` + fe-core `SchemaCacheValue` 在 session=user 下**仍开**,`beginQuerySnapshot` 命中不走 per-user loadTable → 能 list 不能 load 的用户或有 schema/snapshot 泄漏。**定 P4 前须证明无 per-user 授权态,否则按身份/快照分片或走 per-user 委派目录。** + +--- + +## 9. 定稿蓝图（P1 键石，可动工粒度） + +**骨架 = span 派;值 = 忠实 Trino 派的 `CatalogStatementTransaction`;close = 走 query-finish 钩子;键含属主连接器。** + +### 9.1 新增/改动(P1) +- **`ConnectorStatementScope`(fe-connector-api,加类型化方法)**:`ConnectorMetadata getOrCreateMetadata(MetaKey key, Supplier factory)`;`MetaKey = (catalogId, 属主连接器身份)`。`NONE` 每次跑 factory(离线/测试字节不变)。旧 `computeIfAbsent` 迁移期共存。 +- **`ConnectorStatementScopeImpl`(fe-core)**:背 `Map`;`CatalogStatementTransaction` 持 memoized `ConnectorMetadata`(+ P3 持写 `ConnectorTransaction`);提供 `closeAll()`。 +- **funnel(fe-core 静态)**:`PluginDrivenMetadata.get(connector, session)` = `session.getStatementScope().getOrCreateMetadata(key(session,connector), () -> connector.getMetadata(session))`。**~40(实测 ~64)缝**一律改调它替 `connector.getMetadata(session)`(connector+session 每处都在作用域内;扫描节点已把二者存字段,再存 memoized metadata 一字段去掉 per-method 重取)。 +- **`ConnectorMetadata.close()`**:默认 no-op(P2/P3 起连接器 override 释放 Table/FileIO/事务)。 +- **确定性 close**:在**现有 `registerQueryFinishCallback`/`unregisterQuery`** 注册 `scope.closeAll()`(pump 静默后);`resetConnectorStatementScope()` 改为**先 closeAll 再置空**;close 栅栏在 SplitSourceManager 注销之后。 +- **HMS sibling 改道**:`HiveConnectorMetadata` 的 sibling `getMetadata` 经属主连接器身份走同一 funnel(键含属主)→ 每 sibling 一实例。 +- **arch/checkstyle 门禁**:禁 funnel 外直调 `connector.getMetadata`(白名单:funnel 自身 + 明确排除的 off-thread 跨语句 loader)。 + +### 9.2 P1 终态与验证 +- 一条语句一 catalog(一属主连接器)**恰好一个 memoized `ConnectorMetadata`**,读/扫描/写/DDL 全缝复用,pump 静默后确定性 close。连接器内部零改(iceberg 的 `IcebergStatementScope` 共存于 impl,P2 再删)。 +- **P1 字节中性**:NONE 下逐次 factory=今日行为;perf delta≈0(语句内去重本就有)。收益=单漏斗 + 无 per-connector memo + 确定性生命周期。 +- **守门**:①每(语句,catalog)加载计数=1(对照 NONE=N);②跨 catalog/跨 queryId 隔离;③预编译重执行不泄漏(closeAll 被调);④异构 HMS 网关下每 sibling 一实例、加载计数=1;⑤游标取数/转发/重试/SQL-cache 路径 close 恰好一次、不早不晚(针对盲区①的回归)。 + +### 9.3 后续期(定稿方向,细化留各自立项) +- **P2**:once-loaded Table/删除桥/列句柄 → 实例字段(硬 load-once,超越弱点 A);删 `IcebergStatementScope` string-key + paimon 胖句柄。 +- **P3**:写事务并入 `CatalogStatementTransaction`;beginWrite 取实例已解析表;收编两 session。 +- **P4**:元数据/凭证拆分 + 按身份分片授权敏感投影 + 删 `SUPPORTS_USER_SESSION` 全关门禁;先解决 §8.3⑤ 残留泄漏威胁模型。 +- **独立性能项**:`statementSession()` 每语句 memoize session(灭 ~26× 反射 dump);审计所有 build 点确认 session 属性语句内恒定。 + +### 9.4 待你拍板才动代码 +本文件是设计定稿方向。**下一步(若你点头):把 §9.1 细化成 seam-by-seam 的 P1 实现设计(逐文件逐方法),仍不动代码;或直接进入 P1 实现。** 另:P4 的 §8.3⑤ 残留泄漏属安全威胁模型,建议单独走一次签字(涉及"能 list 不能 load"的元数据披露)。 diff --git a/plan-doc/per-statement-table-owner-port/progress.md b/plan-doc/per-statement-table-owner-port/progress.md index f4fe70709c2307..91ffaf040ca11f 100644 --- a/plan-doc/per-statement-table-owner-port/progress.md +++ b/plan-doc/per-statement-table-owner-port/progress.md @@ -26,3 +26,39 @@ - **高度分级 L0/L1/L2/L3**；**paimon-加写=L1 抽共享 helper 的正确触发点**（有 2 个真实用户）；共享 helper 落 `fe-connector-api`（不碰铁律 A）。 - **用户拍板**：先把结论记成**单独文档** → `designs/recon-findings-and-trino-refactor-groundwork.md`；**重构成 Trino 架构（L2/L3）留下个 session 专题讨论**（该文档 §7 已备预备材料）。 - **未动任何产品代码**（纯 plan-doc）。**下一步**：见 HANDOFF —— 下个 session = 讨论"重构成 Trino 架构"的可行性/分期/与铁律 A 的取舍。 + +## 2026-07-19 — session 2：Trino 式重构专题（用户豁免铁律）→ 目标架构定稿方向 + 红队 + +- **用户新指令**：抛开一切铁律/约束，可改任何模块，目标=至少和 Trino 一样合理、能更好则更好。 +- **workflow ①（`wf_72d1e505-75c`）现状+Trino recon + 对抗验证**：更正旧文档——`ConnectorMetadata` 是**每调用即弃外壳**（`Connector` 才是单例），改每语句不用砸单例；句柄已 fact-carrying（Trino invariant 3 已满足）；`StatementContext` 已是被验证的 span 宿主（off-thread 靠构造期捕获）；外部写=每语句 autocommit（1 StatementContext≈1 外部事务）；Trino 四不变量 + 两弱点（iceberg 软 load-once / 冗余双 memo）源码核实。 +- **定稿设计** → `designs/trino-parity-metadata-redesign-design.md`：目标架构=引擎自持每语句 metadata 实例 + 管道层唯一 memo + 下层按身份分片(拆凭证)缓存。 +- **用户拍板**：终点 P1–P4（不含 P5）；直上 P1 键石；语句末确定性 close；先红队。 +- **workflow ②（`wf_62cc379e-c6e`）4 架构师 × 4 评审对抗红队**：胜出=span 派（3/4 轴第一）；嫁接忠实-Trino 的 `CatalogStatementTransaction` + 缓存派"元数据/凭证拆分" + 最小面派若干硬化 + arch 门禁。**揪出五处公共盲区（两硬伤）**：①close 须走现有 `registerQueryFinishCallback` 而非 StatementContext.close；②HMS sibling 扇出须键含属主连接器 + sibling 路由进 funnel；③EXECUTE reset 先 closeAll；④close 栅栏在 SplitSourceManager 注销后；⑤P4 残留泄漏威胁模型待签字。已并入 §8/§9。 +- **未动任何产品代码**（纯 plan-doc）。**下一步**：见 HANDOFF —— 等用户点头后细化 P1 seam-by-seam 实现设计或直接进入 P1 实现。 + +## 2026-07-19 — session 2（续）：P1 seam-by-seam 实现设计（grounding + 定稿） + +- **workflow ③（`wf_7e537094-44f`）只读 grounding**（5 读者：seam 清单 / close 生命周期 / HMS sibling / scope-session SPI / 测试+门禁基建）。核实产出：**66 处真实 seam**（改道表）；close 钩子=`registerQueryFinishCallback`（须移注册点到 scope 创建、closeAll 幂等）；**HMS 网关已 LIVE**（2026-07-17 进 SPI_READY_TYPES，dormant 注释过期，三连接器共享 catalogId→key 加属主 label）；`ConnectorMetadata` 已 Closeable no-op（memoize 安全）；守门/门禁全有模板。 +- **定稿** → `designs/P1-implementation-design.md`：§1 SPI/plumbing 签名、§2 66 缝改道、§3 扫描节点存字段、§4 close 接线、§5 HMS sibling funnel、§6 read-vs-write（写侧 7 缝留 P3）、§7 守门、§8 arch 门禁、§9 四个待确认点、§10 commit 切分 C1–C5。 +- **未动任何产品代码**（纯 plan-doc）。**下一步**：见 HANDOFF —— 等用户确认 §9 四点后进入 P1 实现（C1→C5）。 + +## 2026-07-19 — session 3：扩展范围取证 + 用户二次拍板 → 定稿分期 + +- **用户就 §9 四点拍板，偏离两处原推荐**：写侧要一起做、后台加载点"传句柄到后台线程"、缓存隔离现在就做。 +- **workflow（`wf_8b907b93-e9f`，18 agents，recon+对抗+综合）** 三块结论： + - **读写共用**：可行且忠实 Trino（`CatalogTransaction` 单实例读写共用已核实）；机制=`CatalogStatementTransaction` 共持体 + 写臂 8 处改道；两闸门=按连接器保留起写刷新（hive 留 beginWrite getTable）+ 读写身份指纹相等。 + - **后台传递（用户方向被证伪）**：把实例传后台不安全（共享池 worker 复用 → 已关闭对象被无关查询误用）；纠正=语句派生异步"提交时捕获"+ 7 处跨语句 loader 显式强制 NONE 读穿；**查出并建议修 `fetchRowCount` 在 ANALYZE 线程被错误绑定的真实隐患**。 + - **缓存隔离**：影响面小（仅 iceberg 三投影缓存 + fe-core 表结构缓存）；两轴切分（授权敏感投影按身份分片、含凭证原始表永不跨语句缓存）+ 新 SPI `getIdentityShardKey()`；HMS 异构网关是最尖的洞。 + - **综合**：强烈建议**分期**（硬依赖 + 后台传递被证伪 + 缓存隔离是独立安全工程；一次性全做=不可二分回归面）。 +- **用户二次拍板（动码前）**：①**分期推进**；②后台**采纳纠正做法 + 修隐患**；③**确认 list ≠ load** → 缓存隔离=真实越权修复，独立安全 track。 +- **定稿** → `designs/expanded-scope-phasing-and-security-decisions.md`：决策表 + 三块取证 + 分期 STEP 1（读取键石，含后台纠正）/ STEP 2（HMS 兄弟）/ STEP 3（写入共用，拆 3a/3b）/ STEP 4（缓存隔离，独立安全 track）+ 对旧文档更正。 +- **未动任何产品代码**（纯 plan-doc）。**下一步**：见 HANDOFF —— 等用户对 STEP 1 逐提交方案点头即进入实现（C1→C3 = 键石 + 后台纠正）。 + +## 2026-07-19 — session 3（续）：落地 STEP 1 的 C1 地基 + C2 关闭接线 + +- 用户逐提交确认后开写。**首次改动产品代码**（用户已豁免铁律 A）。 +- **C1 地基**（commit `5b7312f9d1f`）：`ConnectorStatementScope` 加 `getOrCreateMetadata`/`closeAll` 默认方法；`ConnectorStatementScopeImpl.closeAll` 幂等 close-once + best-effort；新增静态漏斗 `PluginDrivenMetadata.get(session, connector)`（按 catalogId memoize `connector.getMetadata`）。fe-core 单测 memo-once/NONE-each/跨目录隔离/关一次。字节中性（无生产路径接进漏斗）。编译 + checkstyle 0 + 8 测全绿。 +- **动手 C2 前的取证**（workflow `wf_9250330b-e81`）：发现 `captureStatementScope` 在**每次 on-thread 会话构建**就急切建 scope；原设计"在 scope 创建处注册关闭回调"**会泄漏**——DDL/`SHOW`/`DESCRIBE`/`EXPLAIN`/前台 `ANALYZE` 走 `Command.run` 建 scope 却永不到 `unregisterQuery`，回调连带 scope 永留无 TTL 注册表。已对抗复核。 +- **C2 关闭接线**（commit `12f3e95239b`）：改为**两层关闭**——主关闭 `PluginDrivenScanNode.getSplits` 注册 `scope::closeAll`（对象捕获、跳 NONE、同 read-txn queryId 键，只对走协调器语句触发、pump 静默后）；兜底 `StatementContext.close()` 的 `isReturnResultFromLocal` 守卫 closeAll（直连 `executeQuery` finally + `proxyExecute` 新增 finally 覆盖转发主节点）；`resetConnectorStatementScope` 改 closeAll-before-null；`handleQueryWithRetry` 每重试重置。核实 `releasePlannerResources` 幂等故转发路径补 close 安全。单测 reset-先关/close-本地关/close-异步延后。P1 关闭仍 no-op，行为中性。编译 + checkstyle 0 + 11 测全绿。 +- 更正 `P1-implementation-design.md` §4（旧"scope 创建处注册"→两层关闭）；残留风险（取消非硬栅栏、arrow-flight 断连、待确认的非-getSplits 外部查询、**TCCL 自钉扎硬前置**）记入分期定稿 §6。 +- **下一步**：见 HANDOFF —— 下个 session 做 C3（读取侧改道 ~55 缝 + 扫描节点存字段 + 7 处后台读穿纠正 + 修 fetchRowCount 隐患），动手前先核行号列清单 + 配对抗复核。用户将在下个 session 开新任务。 diff --git a/plan-doc/per-statement-table-owner-port/tasklist.md b/plan-doc/per-statement-table-owner-port/tasklist.md index 409b1674025072..bf299d7c151baf 100644 --- a/plan-doc/per-statement-table-owner-port/tasklist.md +++ b/plan-doc/per-statement-table-owner-port/tasklist.md @@ -4,7 +4,22 @@ > 立项流程、约束铁律、模板见 [`README.md`](./README.md)。地基（fe-core/api SPI）已就位，移植=纯连接器侧。 > 状态图例：⏳ 待启动 · 🔍 recon 中 · 🚧 进行中 · ✅ 完成 · 🔬 复核判不必做 · ❌ 阻塞 -## 总览 +## ⭐ 现行主线 = Trino 式重构分期（2026-07-19 session 3 定稿） + +> 逐连接器移植（PORT-01~04）经复核**全部 🔬 判暂不做**；用户改为**直接重构成 Trino 式"引擎自持每语句 metadata 实例"架构**（豁免铁律 A）。定稿分期见 `designs/expanded-scope-phasing-and-security-decisions.md`。不砍任何一块，只排序 + 独立提交/验证。 + +| ID | 步 | 状态 | 内容 | 依赖 | commit | +|---|---|---|---|---|---| +| RD-1 | STEP 1 读取键石 | 🚧 进行中（C1+C2 已提交，C3 待做） | C1 地基 ✅ + C2 关闭接线 ✅ + C3 读/扫描/DDL/MVCC 改道 + 扫描节点存字段 + 7 处后台 loader 显式 NONE 读穿 + 修 fetchRowCount ANALYZE 隐患（待做） | — | C1 `5b7312f9d1f` · C2 `12f3e95239b` | +| RD-2 | STEP 2 HMS 兄弟扇出 | ⏳ 待启动 | 键=(catalogId, ownerLabel)；兄弟 getMetadata **及 beginTransaction** 进同一 funnel；异构网关 e2e | RD-1 | | +| RD-3 | STEP 3 写入共用 | ⏳ 待启动 | 3a 无状态写点改道进 funnel；3b `ConnectorTransaction` 归属上移 `CatalogStatementTransaction` + commit/rollback vs closeAll 顺序。闸门=读写身份等价 + 保留 hive 起写刷新 + 保留 tx↔session 绑定 | RD-1,RD-2 | | +| RD-4 | STEP 4 缓存隔离（安全 track） | ⏳ 待启动（**list≠load 已确认=真实越权**） | `getIdentityShardKey()` SPI → iceberg 三投影缓存 Key 分片 + fe-core 表结构缓存 bypass(先)/分片(后) + 防漂移门禁；随 STEP 2 属主键覆盖异构网关；越权 e2e | 威胁模型签字 + RD-2 | | + +> **纠缠点（RD-3b 时定）**：iceberg 起写复用表读 `IcebergStatementScope.sharedTable`，该 side-car 在 P2 计划要删 → 先接旧 vs 先做 iceberg 最小 P2 前置。 + +--- + +## 总览（历史 · 逐连接器移植框架，已被重构主线取代） - **蓝本 = iceberg（PERF-07 已完成）**：`ConnectorStatementScope`(fe-core/api，中性) + `IcebergStatementScope`(连接器) + 四处 `resolveTable*` 共享 + 拆胖句柄 + 删除暂存下沉 + v3 fail-loud。commits `97bdcd6bdbe`(fe-core) + `ea7fd1f6e7a`(iceberg)。 - **本清单只跟踪"其它连接器"的移植**；每项**第一步是 recon**（可能复核判"不必做"）。 From d6de751e419ed22e1e3e6e30f62b7d0598aa4250 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 19:37:00 +0800 Subject: [PATCH 32/47] [feat](catalog) fe-core: route read/scan/DDL/MVCC through the per-statement metadata funnel + force read-through for cross-statement loaders Completes the read-side of the engine-held per-statement ConnectorMetadata keystone (C1 funnel + C2 close wiring already landed). Four mechanical parts, all in one commit so every intermediate state stays compilable and green (the sub-parts interleave within two shared files): 1. buildCrossStatementSession() helper on PluginDrivenExternalCatalog: mirrors buildConnectorSession() (same delegated-credential handling) but forces the per-statement scope to ConnectorStatementScope.NONE. Under NONE the funnel memoizes nothing, so a metadata resolved with this session is built fresh and never bound into -- nor closed with -- a live statement's scope. 2. Force read-through for the 9 cross-statement background loaders (database / table name caches, schema cache, column-statistic cache, chunk-size and row-count loaders, name-mapping resolvers, and the BE-driven metadata TVF): each now builds its session via buildCrossStatementSession() and resolves through PluginDrivenMetadata.get(...). This also fixes a latent hazard where fetchRowCount, reached synchronously from AnalysisManager.buildAnalysisJobInfo on the ANALYZE execution thread, would otherwise capture that statement's live scope; forcing NONE makes read-through a contract rather than an accident. 3. Reroute the 49 on-thread read / DDL / command-TVF / MVCC seams from connector.getMetadata(session) to PluginDrivenMetadata.get(session, connector), so a statement resolves one memoized ConnectorMetadata per (catalog) instead of rebuilding it at each resolver. 4. PluginDrivenScanNode: add a volatile cachedMetadata field + metadata() accessor (mirroring the existing resolvedScanProvider holder); the 8 per-method resolvers share it, and the static create() factory routes through the funnel directly. The write-path getMetadata seams (translator x2, BindSink x2, IcebergMergeSink, IcebergRowLevelDmlTransform, InsertExecutor, and the write-only resolveWriteTargetHandle) are deliberately left unchanged for the write-sharing step. Tests: adapt the plugin/scan/mvcc/command/TVF unit tests -- their mocked ConnectorSession now flows through the funnel, so getStatementScope() must return NONE (a Mockito mock returns null for the default method), and test doubles that override buildConnectorSession() also override buildCrossStatementSession(). No assertions or verify() call-counts were weakened. Adds ConnectorSessionImplTest.explicitNoneStatementScopeWinsOverLiveContext pinning that an explicit NONE scope wins over a live ConnectContext capture. Verification: fe-core main compile BUILD SUCCESS; 265 targeted tests green; checkstyle 0 violations. Byte-neutral under NONE (offline/no ConnectContext), matching pre-funnel behavior. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../mvcc/PluginDrivenMvccExternalTable.java | 7 +- .../plugin/PluginDrivenExternalCatalog.java | 89 +++++++++++++------ .../plugin/PluginDrivenExternalDatabase.java | 2 +- .../plugin/PluginDrivenExternalTable.java | 40 ++++----- .../datasource/scan/PluginDrivenScanNode.java | 35 ++++++-- .../plans/commands/ShowPartitionsCommand.java | 3 +- .../commands/call/CallExecuteStmtFunc.java | 3 +- .../execute/ConnectorExecuteAction.java | 3 +- .../JdbcQueryTableValueFunction.java | 3 +- .../tablefunction/MetadataGenerator.java | 5 +- .../connector/ConnectorSessionImplTest.java | 34 +++++++ .../datasource/PluginDrivenSysTableTest.java | 17 ++++ .../PluginDrivenMvccExternalTableTest.java | 9 ++ ...inDrivenExternalCatalogDdlRoutingTest.java | 7 ++ .../PluginDrivenExternalDatabaseTest.java | 3 + ...luginDrivenExternalTablePartitionTest.java | 21 ++++- ...PluginDrivenExternalTableRowCountTest.java | 8 ++ .../plugin/PluginDrivenExternalTableTest.java | 38 +++++++- .../PluginDrivenScanNodeSysHandleTest.java | 8 ++ ...ShowPartitionsCommandPluginDrivenTest.java | 5 ++ .../execute/ConnectorExecuteActionTest.java | 9 ++ .../MetadataGeneratorPluginDrivenTest.java | 5 ++ 22 files changed, 282 insertions(+), 72 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java index 2cdd3ecc04ea77..bb3edc19f44fad 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java @@ -45,6 +45,7 @@ import org.apache.doris.datasource.SchemaCacheValue; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; import org.apache.doris.datasource.plugin.PluginDrivenExternalTable; +import org.apache.doris.datasource.plugin.PluginDrivenMetadata; import org.apache.doris.datasource.plugin.PluginDrivenSchemaCacheValue; import org.apache.doris.mtmv.MTMVBaseTableIf; import org.apache.doris.mtmv.MTMVMaxTimestampSnapshot; @@ -140,7 +141,7 @@ private PluginDrivenMvccSnapshot materializeLatest() { Collections.emptyMap(), Collections.emptyMap()); } ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { @@ -355,7 +356,7 @@ public MvccSnapshot loadSnapshot(Optional tableSnapshot, Optional PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; Connector connector = pluginCatalog.getConnector(); ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); String dbName = db != null ? db.getRemoteName() : ""; String tableName = getRemoteName(); Optional handleOpt = resolveConnectorTableHandle(session, metadata); @@ -722,7 +723,7 @@ private Optional resolveFreshnessProbe() { return Optional.empty(); } ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); return handleOpt.map(handle -> new FreshnessProbe(session, metadata, handle)); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java index 0985a6e795632c..6f53ff33fac7f6 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java @@ -40,6 +40,7 @@ import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorTestResult; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.ddl.ConnectorColumnPosition; @@ -291,8 +292,8 @@ public void onRefreshCache(boolean invalidCache) { @Override protected List listDatabaseNames() { try { - ConnectorSession session = buildConnectorSession(); - return connector.getMetadata(session).listDatabaseNames(session); + ConnectorSession session = buildCrossStatementSession(); + return PluginDrivenMetadata.get(session, connector).listDatabaseNames(session); } catch (RuntimeException e) { // The connector connects lazily: initLocalObjectsImpl() only constructs it, so the // first metastore round-trip happens here — inside the meta-cache loader, which runs @@ -307,8 +308,8 @@ protected List listDatabaseNames() { @Override protected List listTableNamesFromRemote(SessionContext ctx, String dbName) { - ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorSession session = buildCrossStatementSession(); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); List tableNames = metadata.listTableNames(session, dbName); if (!connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_VIEW)) { return tableNames; @@ -329,7 +330,7 @@ protected List listTableNamesFromRemote(SessionContext ctx, String dbNam @Override public boolean tableExist(SessionContext ctx, String dbName, String tblName) { ConnectorSession session = buildConnectorSession(); - return connector.getMetadata(session) + return PluginDrivenMetadata.get(session, connector) .getTableHandle(session, dbName, tblName).isPresent(); } @@ -419,7 +420,7 @@ public boolean createTable(CreateTableInfo createTableInfo) throws UserException + "' in catalog: " + getName()); } ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); // Mirror legacy MaxComputeMetadataOps.createTableImpl:178-197 -- probe BOTH the remote // (connector) and the local FE cache for an existing table. On IF NOT EXISTS this lets CTAS // short-circuit (Env.createTable contract: return true when the table already exists), so a @@ -498,7 +499,7 @@ public void createDb(String dbName, boolean ifNotExists, Map pro return; } ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); // FE-cache miss but the db may already exist REMOTELY (created on another FE / before this // FE's db-name cache was populated). Legacy MaxComputeMetadataOps.createDbImpl consulted // BOTH getDbNullable AND the remote databaseExist, and IF NOT EXISTS then no-oped. Mirror @@ -547,7 +548,7 @@ public void dropDb(String dbName, boolean ifExists, boolean force) throws DdlExc } ConnectorSession session = buildConnectorSession(); try { - connector.getMetadata(session).dropDatabase(session, db.getRemoteName(), ifExists, force); + PluginDrivenMetadata.get(session, connector).dropDatabase(session, db.getRemoteName(), ifExists, force); } catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } @@ -595,7 +596,7 @@ public void dropTable(String dbName, String tableName, boolean isView, boolean i throw new DdlException("Failed to get table: '" + tableName + "' in database: " + dbName); } ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); // Route a DROP on a VIEW to dropView, mirroring legacy IcebergMetadataOps.dropTableImpl's // viewExists -> performDropView dispatch: a connector that exposes views keeps them in a separate // namespace, so getTableHandle/tableExists below is false for a view and the table-handle path @@ -667,7 +668,7 @@ public void renameTable(String dbName, String oldTableName, String newTableName) throw new DdlException("Failed to get table: '" + oldTableName + "' in database: " + dbName); } ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(dorisTable, session, metadata); try { metadata.renameTable(session, handle, newTableName); @@ -711,7 +712,7 @@ public void truncateTable(String dbName, String tableName, PartitionNamesInfo pa } List partitions = partitionNamesInfo == null ? null : partitionNamesInfo.getPartitionNames(); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(dorisTable, session, metadata); try { metadata.truncateTable(session, handle, partitions); @@ -804,7 +805,7 @@ public void replayCreateTable(String dbName, String tblName) { public void addColumn(TableIf dorisTable, Column column, ColumnPosition position) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -820,7 +821,7 @@ public void addColumn(TableIf dorisTable, Column column, ColumnPosition position public void addColumns(TableIf dorisTable, List columns) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -835,7 +836,7 @@ public void addColumns(TableIf dorisTable, List columns) throws UserExce public void dropColumn(TableIf dorisTable, String columnName) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -850,7 +851,7 @@ public void dropColumn(TableIf dorisTable, String columnName) throws UserExcepti public void renameColumn(TableIf dorisTable, String oldName, String newName) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -865,7 +866,7 @@ public void renameColumn(TableIf dorisTable, String oldName, String newName) thr public void modifyColumn(TableIf dorisTable, Column column, ColumnPosition position) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -881,7 +882,7 @@ public void modifyColumn(TableIf dorisTable, Column column, ColumnPosition posit public void reorderColumns(TableIf dorisTable, List newOrder) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -912,7 +913,7 @@ public void createOrReplaceBranch(TableIf dorisTable, CreateOrReplaceBranchInfo throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -928,7 +929,7 @@ public void createOrReplaceBranch(TableIf dorisTable, CreateOrReplaceBranchInfo public void createOrReplaceTag(TableIf dorisTable, CreateOrReplaceTagInfo tagInfo) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -944,7 +945,7 @@ public void createOrReplaceTag(TableIf dorisTable, CreateOrReplaceTagInfo tagInf public void dropBranch(TableIf dorisTable, DropBranchInfo branchInfo) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -960,7 +961,7 @@ public void dropBranch(TableIf dorisTable, DropBranchInfo branchInfo) throws Use public void dropTag(TableIf dorisTable, DropTagInfo tagInfo) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -985,7 +986,7 @@ public void dropTag(TableIf dorisTable, DropTagInfo tagInfo) throws UserExceptio public void addPartitionField(TableIf dorisTable, AddPartitionFieldOp op) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -1000,7 +1001,7 @@ public void addPartitionField(TableIf dorisTable, AddPartitionFieldOp op) throws public void dropPartitionField(TableIf dorisTable, DropPartitionFieldOp op) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -1015,7 +1016,7 @@ public void dropPartitionField(TableIf dorisTable, DropPartitionFieldOp op) thro public void replacePartitionField(TableIf dorisTable, ReplacePartitionFieldOp op) throws UserException { ExternalTable externalTable = checkExternalTable(dorisTable); ConnectorSession session = buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle handle = resolveAlterHandle(externalTable, session, metadata); long updateTime = System.currentTimeMillis(); try { @@ -1102,14 +1103,15 @@ protected void afterExternalRename(String dbName, String oldTableName, String ne @Override public String fromRemoteDatabaseName(String remoteDatabaseName) { - ConnectorSession session = buildConnectorSession(); - return connector.getMetadata(session).fromRemoteDatabaseName(session, remoteDatabaseName); + ConnectorSession session = buildCrossStatementSession(); + return PluginDrivenMetadata.get(session, connector).fromRemoteDatabaseName(session, remoteDatabaseName); } @Override public String fromRemoteTableName(String remoteDatabaseName, String remoteTableName) { - ConnectorSession session = buildConnectorSession(); - return connector.getMetadata(session).fromRemoteTableName(session, remoteDatabaseName, remoteTableName); + ConnectorSession session = buildCrossStatementSession(); + return PluginDrivenMetadata.get(session, connector) + .fromRemoteTableName(session, remoteDatabaseName, remoteTableName); } /** @@ -1136,6 +1138,37 @@ public ConnectorSession buildConnectorSession() { .build(); } + /** + * Builds a {@link ConnectorSession} for a CROSS-STATEMENT background loader — one that fills a cache + * living longer than any single statement (database/table name caches, schema cache, column-statistic + * cache, row-count cache, the BE-driven metadata TVF). Identical to {@link #buildConnectorSession()} + * (same credential handling) except the per-statement scope is forced to + * {@link ConnectorStatementScope#NONE}. That makes the read-through a contract rather than an accident: + * a metadata resolved through {@link PluginDrivenMetadata#get} with this session is built fresh and never + * memoized into — nor closed with — some live statement's scope, even when the loader happens to run on a + * request/ANALYZE thread that has one (e.g. {@code fetchRowCount} reached synchronously from + * {@code AnalysisManager.buildAnalysisJobInfo}). Under NONE the funnel memoizes nothing, so this is + * byte-identical to a bare {@code getMetadata} call. + */ + public ConnectorSession buildCrossStatementSession() { + ConnectContext ctx = ConnectContext.get(); + if (ctx != null) { + return ConnectorSessionBuilder.from(ctx) + .withCatalogId(getId()) + .withCatalogName(getName()) + .withCatalogProperties(catalogProperty.getProperties()) + .withUserSessionCapability(supportsUserSession()) + .withStatementScope(ConnectorStatementScope.NONE) + .build(); + } + return ConnectorSessionBuilder.create() + .withCatalogId(getId()) + .withCatalogName(getName()) + .withCatalogProperties(catalogProperty.getProperties()) + .withStatementScope(ConnectorStatementScope.NONE) + .build(); + } + /** * Whether the backing connector projects the querying user's delegated credential onto the remote * metadata source ({@link ConnectorCapability#SUPPORTS_USER_SESSION}), gating both the FE credential diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabase.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabase.java index a8ccb7515049e7..324d72ab886d95 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabase.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabase.java @@ -81,7 +81,7 @@ public String getLocation() { return ""; } ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorDatabaseMetadata dbMetadata = metadata.getDatabase(session, getRemoteName()); return dbMetadata.getProperties().getOrDefault(ConnectorDatabaseMetadata.LOCATION_PROPERTY, ""); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java index 56d056ce7142f0..b6dd3373fecc84 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java @@ -156,7 +156,7 @@ public List getSyntheticScanPredicates(MvccSnapshot snapsho ConnectorMvccSnapshot connectorSnapshot = ((PluginDrivenMvccSnapshot) snapshot).getConnectorSnapshot(); PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = pluginCatalog.getConnector().getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, pluginCatalog.getConnector()); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { return Collections.emptyList(); @@ -186,7 +186,7 @@ public boolean supportsParallelWrite() { */ private Optional resolveWriteCapabilityHandle(Connector connector) { ConnectorSession session = ((PluginDrivenExternalCatalog) catalog).buildConnectorSession(); - return resolveConnectorTableHandle(session, connector.getMetadata(session)); + return resolveConnectorTableHandle(session, PluginDrivenMetadata.get(session, connector)); } /** @@ -456,8 +456,8 @@ public Optional initSchema() { } } Connector connector = pluginCatalog.getConnector(); - ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorSession session = pluginCatalog.buildCrossStatementSession(); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); String dbName = db != null ? db.getRemoteName() : ""; String tableName = getRemoteName(); @@ -573,7 +573,7 @@ protected boolean resolveIsView() { return false; } ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); String dbName = db != null ? db.getRemoteName() : ""; return metadata.viewExists(session, dbName, getRemoteName()); } @@ -591,7 +591,7 @@ public String getViewText() { PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; Connector connector = pluginCatalog.getConnector(); ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); String dbName = db != null ? db.getRemoteName() : ""; ConnectorViewDefinition definition = metadata.getViewDefinition(session, dbName, getRemoteName()); return definition.getSql(); @@ -614,7 +614,7 @@ public Optional getShowCreateTableDdl() { return Optional.empty(); } ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { return Optional.empty(); @@ -724,7 +724,7 @@ private List fetchSyntheticWriteColumns() { // so a read-only connector now resolves the handle here — a no-op for a write-capable connector, which // resolved it regardless. ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { return Collections.emptyList(); @@ -817,7 +817,7 @@ public Map getNameToPartitionItems(Optional PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; Connector connector = pluginCatalog.getConnector(); ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { return Collections.emptyMap(); @@ -878,7 +878,7 @@ public Map> getNameToPartitionValues(Optional PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; Connector connector = pluginCatalog.getConnector(); ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { return Collections.emptyMap(); @@ -915,7 +915,7 @@ public List> getMetadataTableRows(String kind) { return Collections.emptyList(); } ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { return Collections.emptyList(); @@ -947,7 +947,7 @@ public String getComment(boolean escapeQuota) { PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; Connector connector = pluginCatalog.getConnector(); ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); String tableName = getRemoteName(); String comment = metadata.getTableComment(session, remoteDbName, tableName); if (escapeQuota && comment != null) { @@ -978,7 +978,7 @@ public Map getSupportedSysTables() { PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; Connector connector = pluginCatalog.getConnector(); ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { return Collections.emptyMap(); @@ -1048,8 +1048,8 @@ public Optional getColumnStatistic(String colName) { if (connector == null) { return Optional.empty(); } - ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorSession session = pluginCatalog.buildCrossStatementSession(); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { return Optional.empty(); @@ -1083,8 +1083,8 @@ public List getChunkSizes() { if (connector == null) { return Collections.emptyList(); } - ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorSession session = pluginCatalog.buildCrossStatementSession(); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { return Collections.emptyList(); @@ -1149,8 +1149,8 @@ public long fetchRowCount() { makeSureInitialized(); PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; Connector connector = pluginCatalog.getConnector(); - ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorSession session = pluginCatalog.buildCrossStatementSession(); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { @@ -1301,7 +1301,7 @@ public TTableDescriptor toThrift() { PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; Connector connector = pluginCatalog.getConnector(); ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); String dbName = db != null ? db.getRemoteName() : ""; List schema = getFullSchema(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java index adb30c8fed359a..d70715a0342163 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/PluginDrivenScanNode.java @@ -57,6 +57,7 @@ import org.apache.doris.datasource.mvcc.PluginDrivenMvccSnapshot; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; import org.apache.doris.datasource.plugin.PluginDrivenExternalTable; +import org.apache.doris.datasource.plugin.PluginDrivenMetadata; import org.apache.doris.datasource.plugin.PluginDrivenSysExternalTable; import org.apache.doris.datasource.split.FileSplit; import org.apache.doris.datasource.split.PluginDrivenSplit; @@ -182,6 +183,11 @@ public class PluginDrivenScanNode extends FileQueryScanNode { // Volatile so the concurrent partition-batch appendBatch threads read a self-consistent (handle, provider). private volatile ResolvedScanProvider resolvedScanProvider; + // This node's per-statement ConnectorMetadata. The funnel already memoizes it in the statement scope + // (keyed by catalogId); cached here as well so the per-method resolvers don't re-hit the scope map. + // Volatile mirrors resolvedScanProvider — keeps an off-path metadata() read race-free. + private volatile ConnectorMetadata cachedMetadata; + public PluginDrivenScanNode(PlanNodeId id, TupleDescriptor desc, boolean needCheckColumnPriv, SessionVariable sv, ScanContext scanContext, Connector connector, @@ -192,6 +198,17 @@ public PluginDrivenScanNode(PlanNodeId id, TupleDescriptor desc, this.currentHandle = tableHandle; } + // Lazily resolves this node's ConnectorMetadata through the per-statement funnel and caches it, so the + // per-method resolvers below share one instance for the statement instead of rebuilding it each time. + private ConnectorMetadata metadata() { + ConnectorMetadata m = cachedMetadata; + if (m == null) { + m = PluginDrivenMetadata.get(connectorSession, connector); + cachedMetadata = m; + } + return m; + } + /** * Creates a PluginDrivenScanNode by resolving the connector, session, and table handle * from the plugin-driven catalog and table. @@ -202,7 +219,7 @@ public static PluginDrivenScanNode create(PlanNodeId id, TupleDescriptor desc, PluginDrivenExternalTable table) { Connector connector = catalog.getConnector(); ConnectorSession session = catalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); String dbName = table.getDb() != null ? table.getDb().getRemoteName() : ""; // Resolve through the table's sys-aware seam (NOT raw metadata.getTableHandle): for a normal // table this is identical to getTableHandle(session, dbName, remoteName), but for a @@ -802,7 +819,7 @@ protected void convertPredicate() { if (conjuncts == null || conjuncts.isEmpty()) { return; } - ConnectorMetadata metadata = connector.getMetadata(connectorSession); + ConnectorMetadata metadata = metadata(); ConnectorFilterConstraint constraint = buildFilterConstraint(conjuncts); Optional> result = metadata.applyFilter(connectorSession, currentHandle, constraint); @@ -842,7 +859,7 @@ private void tryPushDownLimit() { if (limit <= 0) { return; } - ConnectorMetadata metadata = connector.getMetadata(connectorSession); + ConnectorMetadata metadata = metadata(); Optional> result = metadata.applyLimit(connectorSession, currentHandle, limit); if (result.isPresent()) { @@ -861,7 +878,7 @@ private void tryPushDownProjection(List columns) { if (columns.isEmpty()) { return; } - ConnectorMetadata metadata = connector.getMetadata(connectorSession); + ConnectorMetadata metadata = metadata(); Optional> result = metadata.applyProjection(connectorSession, currentHandle, columns); if (result.isPresent()) { @@ -906,7 +923,7 @@ public static ConnectorTableHandle applyMvccSnapshotPin(ConnectorMetadata metada * split path and the serialized-table path read at the pinned snapshot. */ private void pinMvccSnapshot() throws UserException { - ConnectorMetadata metadata = connector.getMetadata(connectorSession); + ConnectorMetadata metadata = metadata(); // Version-aware lookup: a statement mixing main and @branch/@tag (or FOR-TIME) of the SAME table // pins one snapshot per reference; resolve THIS scan's reference by its own selector so it reads // its own snapshot (not whichever reference loaded first). getQueryTableSnapshot()/getScanParams() @@ -1051,7 +1068,7 @@ private void pinRewriteFileScope() { if (scope == null || scope.isEmpty()) { return; } - ConnectorMetadata metadata = connector.getMetadata(connectorSession); + ConnectorMetadata metadata = metadata(); currentHandle = applyRewriteFileScopePin(metadata, connectorSession, currentHandle, scope); } @@ -1070,7 +1087,7 @@ private void pinTopnLazyMaterialize() { if (!hasTopnLazyMaterializeSlot(desc.getSlots())) { return; } - ConnectorMetadata metadata = connector.getMetadata(connectorSession); + ConnectorMetadata metadata = metadata(); currentHandle = metadata.applyTopnLazyMaterialization(connectorSession, currentHandle); } @@ -1898,7 +1915,7 @@ private static TFileFormatType mapFileFormatType(String format) { * enabling optimized column selection (e.g., SELECT col1, col2 instead of SELECT *). */ private List buildColumnHandles() { - ConnectorMetadata metadata = connector.getMetadata(connectorSession); + ConnectorMetadata metadata = metadata(); Map allHandles = metadata.getColumnHandles(connectorSession, currentHandle); if (allHandles.isEmpty()) { @@ -1935,7 +1952,7 @@ private Optional buildRemainingFilter() { return Optional.empty(); } List pushableConjuncts = conjuncts; - ConnectorMetadata metadata = connector.getMetadata(connectorSession); + ConnectorMetadata metadata = metadata(); if (!metadata.supportsCastPredicatePushdown(connectorSession)) { filteredToOriginalIndex = new ArrayList<>(); pushableConjuncts = new ArrayList<>(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java index 4e6643ba7b0c6b..28d6f0cf525739 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommand.java @@ -46,6 +46,7 @@ import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; import org.apache.doris.datasource.plugin.PluginDrivenExternalTable; +import org.apache.doris.datasource.plugin.PluginDrivenMetadata; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.analyzer.UnboundSlot; import org.apache.doris.nereids.properties.OrderKey; @@ -282,7 +283,7 @@ private ShowResultSet handleShowPluginDrivenTablePartitions() throws AnalysisExc // Route partition listing through the connector SPI. ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = pluginCatalog.getConnector().getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, pluginCatalog.getConnector()); ConnectorTableHandle handle = metadata .getTableHandle(session, dorisTable.getRemoteDbName(), dorisTable.getRemoteName()) .orElseThrow(() -> new AnalysisException( diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/call/CallExecuteStmtFunc.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/call/CallExecuteStmtFunc.java index 6cda00f7425947..6f7624795635bc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/call/CallExecuteStmtFunc.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/call/CallExecuteStmtFunc.java @@ -23,6 +23,7 @@ import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; +import org.apache.doris.datasource.plugin.PluginDrivenMetadata; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.exceptions.AnalysisException; import org.apache.doris.nereids.trees.expressions.Expression; @@ -98,7 +99,7 @@ public void run() { if (catalogIf instanceof PluginDrivenExternalCatalog) { PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalogIf; ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = pluginCatalog.getConnector().getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, pluginCatalog.getConnector()); metadata.executeStmt(session, stmt); } else { throw new AnalysisException("executeStmt not supported for catalog type: " diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java index 6ceb8acabd70d9..2db11166bbc635 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java @@ -43,6 +43,7 @@ import org.apache.doris.datasource.connector.converter.UnboundExpressionToConnectorPredicateConverter; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; import org.apache.doris.datasource.plugin.PluginDrivenExternalTable; +import org.apache.doris.datasource.plugin.PluginDrivenMetadata; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.trees.expressions.Expression; import org.apache.doris.qe.CommonResultSet; @@ -125,7 +126,7 @@ public ResultSet execute(TableIf ignored) throws UserException { // (single-call and distributed) also need the session/handle. Resolving the handle first means a bad // table name now surfaces "Table not found" before "does not support EXECUTE". ConnectorSession session = catalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, connector); ConnectorTableHandle tableHandle = metadata .getTableHandle(session, table.getRemoteDbName(), table.getRemoteName()) .orElseThrow(() -> new AnalysisException("Table not found: " + table.getRemoteDbName() diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/JdbcQueryTableValueFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/JdbcQueryTableValueFunction.java index aa0c4293678b68..f8250bee8a84b5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/JdbcQueryTableValueFunction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/JdbcQueryTableValueFunction.java @@ -26,6 +26,7 @@ import org.apache.doris.connector.api.handle.PassthroughQueryTableHandle; import org.apache.doris.datasource.connector.converter.ConnectorColumnConverter; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; +import org.apache.doris.datasource.plugin.PluginDrivenMetadata; import org.apache.doris.datasource.scan.PluginDrivenScanNode; import org.apache.doris.planner.PlanNodeId; import org.apache.doris.planner.ScanContext; @@ -49,7 +50,7 @@ public JdbcQueryTableValueFunction(Map params) throws AnalysisEx public List getTableColumns() throws AnalysisException { PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalogIf; ConnectorSession session = pluginCatalog.buildConnectorSession(); - ConnectorMetadata metadata = pluginCatalog.getConnector().getMetadata(session); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, pluginCatalog.getConnector()); ConnectorTableSchema schema = metadata.getColumnsFromQuery(session, query); return ConnectorColumnConverter.convertColumns(schema.getColumns()); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java index 277b1095a8d1da..be08a36b69f4cb 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java @@ -72,6 +72,7 @@ import org.apache.doris.datasource.mvcc.MvccUtil; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; import org.apache.doris.datasource.plugin.PluginDrivenExternalTable; +import org.apache.doris.datasource.plugin.PluginDrivenMetadata; import org.apache.doris.job.common.JobType; import org.apache.doris.job.extensions.insert.streaming.AbstractStreamingTask; import org.apache.doris.job.extensions.insert.streaming.StreamingInsertJob; @@ -1325,8 +1326,8 @@ private static TFetchSchemaTableDataResult partitionMetadataResult(TMetadataTabl private static TFetchSchemaTableDataResult dealPluginDrivenCatalog(PluginDrivenExternalCatalog catalog, ExternalTable table) { List dataBatch = Lists.newArrayList(); - ConnectorSession session = catalog.buildConnectorSession(); - ConnectorMetadata metadata = catalog.getConnector().getMetadata(session); + ConnectorSession session = catalog.buildCrossStatementSession(); + ConnectorMetadata metadata = PluginDrivenMetadata.get(session, catalog.getConnector()); Optional handle = metadata.getTableHandle( session, table.getRemoteDbName(), table.getRemoteName()); if (handle.isPresent()) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java index 8e61d5f8e1b707..e745aa6af8d289 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorSessionImplTest.java @@ -19,7 +19,10 @@ import org.apache.doris.connector.api.ConnectorDelegatedCredential; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.nereids.StatementContext; +import org.apache.doris.qe.ConnectContext; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -273,6 +276,37 @@ public void capableConnectorWithNoOfferedCredentialCarriesNone() { Assertions.assertFalse(session.getDelegatedCredential().isPresent()); } + @Test + public void explicitNoneStatementScopeWinsOverLiveContext() { + // A cross-statement background loader (PluginDrivenExternalCatalog#buildCrossStatementSession) forces the + // per-statement scope to NONE so a metadata it resolves is never memoized into — nor closed with — the + // live statement's scope, even when the loader runs on a thread that has one (e.g. fetchRowCount reached + // synchronously from AnalysisManager.buildAnalysisJobInfo on the ANALYZE execution thread). This pins the + // builder guarantee that helper relies on: an explicit withStatementScope(NONE) wins over the scope + // captured from the live ConnectContext. MUTATION: capture ignoring the explicit override -> the loader + // binds to the live statement scope (the leak) -> red. + ConnectContext ctx = new ConnectContext(); + ctx.setThreadLocalInfo(); + try { + StatementContext stmtCtx = new StatementContext(); + ctx.setStatementContext(stmtCtx); + ConnectorStatementScope live = stmtCtx.getOrCreateConnectorStatementScope(); + + // A default session capture binds to the live statement scope (the path a loader must avoid). + ConnectorSession bound = ConnectorSessionBuilder.from(ctx).withCatalogId(1L).build(); + Assertions.assertSame(live, bound.getStatementScope(), + "a default session capture binds to the live statement scope"); + + // The forced-NONE session (what buildCrossStatementSession builds) wins over the live scope. + ConnectorSession crossStatement = ConnectorSessionBuilder.from(ctx).withCatalogId(1L) + .withStatementScope(ConnectorStatementScope.NONE).build(); + Assertions.assertSame(ConnectorStatementScope.NONE, crossStatement.getStatementScope(), + "an explicit NONE scope wins over the live statement context (forced read-through)"); + } finally { + ConnectContext.remove(); + } + } + /** Minimal hand-written {@link ConnectorTransaction}; only identity matters for this test. */ private static final class StubConnectorTransaction implements ConnectorTransaction { private final long txnId; diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java index c396d580551aaa..94a0fe54313f69 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java @@ -22,6 +22,7 @@ import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.handle.ConnectorTableHandle; @@ -65,6 +66,7 @@ public class PluginDrivenSysTableTest { public void testGetSupportedSysTablesDelegatesToConnector() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle baseHandle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("paimon", metadata, session); ExternalDatabase db = mockDb("REMOTE_DB"); @@ -91,6 +93,7 @@ public void testGetSupportedSysTablesDelegatesToConnector() { public void testGetSupportedSysTablesMapsTvfKindToPartitionsSysTable() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle baseHandle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("hms", metadata, session); Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) @@ -128,6 +131,7 @@ public void testGetSupportedSysTablesMapsTvfKindToPartitionsSysTable() { public void testGetSupportedSysTablesEmptyWhenNoBaseHandle() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); TestablePluginCatalog catalog = new TestablePluginCatalog("paimon", metadata, session); Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) .thenReturn(Optional.empty()); @@ -145,6 +149,7 @@ public void testGetSupportedSysTablesEmptyWhenNoBaseHandle() { public void testFindSysTableResolvesBySuffix() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle baseHandle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("paimon", metadata, session); Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) @@ -169,6 +174,7 @@ public void testFindSysTableResolvesBySuffix() { public void testCreateSysExternalTableReportsPluginTypeAndName() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle baseHandle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("paimon", metadata, session); Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) @@ -208,6 +214,7 @@ public void testSysTableThreadsSysHandleNotBaseHandle() { // query reads the system table, not the base. This is the whole point of T18. ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle baseHandle = Mockito.mock(ConnectorTableHandle.class); ConnectorTableHandle sysHandle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("paimon", metadata, session); @@ -252,6 +259,7 @@ protected synchronized void makeSureInitialized() { public void testSysTableEmptyWhenBaseHandleMissing() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); TestablePluginCatalog catalog = new TestablePluginCatalog("paimon", metadata, session); Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) .thenReturn(Optional.empty()); @@ -276,6 +284,7 @@ protected synchronized void makeSureInitialized() { public void sysExternalTableReportsBaseTableMysqlTypeMatchingLegacyIceberg() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); TestablePluginCatalog catalog = new TestablePluginCatalog("iceberg", metadata, session); PluginDrivenExternalTable base = bareTable(catalog, mockDb("REMOTE_DB"), "REMOTE_TBL"); PluginDrivenSysExternalTable sys = new PluginDrivenSysExternalTable(base, "snapshots"); @@ -294,6 +303,7 @@ public void sysExternalTableReportsBaseTableMysqlTypeMatchingLegacyIceberg() { public void sysExternalTableReportsIcebergEngineAndEngineTableTypeName() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); TestablePluginCatalog catalog = new TestablePluginCatalog("iceberg", metadata, session); PluginDrivenExternalTable base = bareTable(catalog, mockDb("REMOTE_DB"), "REMOTE_TBL"); PluginDrivenSysExternalTable sys = new PluginDrivenSysExternalTable(base, "snapshots"); @@ -316,6 +326,7 @@ public void sysExternalTableReportsIcebergEngineAndEngineTableTypeName() { public void positionDeletesAbsentWhenConnectorDoesNotListIt() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle baseHandle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("iceberg", metadata, session); Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) @@ -349,6 +360,7 @@ public void positionDeletesAbsentWhenConnectorDoesNotListIt() { public void sysExternalTableIsTransientNeitherRegisteredNorSerialized() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle baseHandle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("iceberg", metadata, session); Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) @@ -434,6 +446,11 @@ public ConnectorSession buildConnectorSession() { return session; } + @Override + public ConnectorSession buildCrossStatementSession() { + return buildConnectorSession(); + } + @Override protected List listDatabaseNames() { return Collections.emptyList(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTableTest.java index f36542c84d41ad..ffbdff5e476569 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTableTest.java @@ -32,6 +32,7 @@ import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.handle.ConnectorTableHandle; @@ -394,6 +395,7 @@ public void testNoCacheReadsFreshSchemaElseCached() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); TestablePluginCatalog catalog = new TestablePluginCatalog(metadata, session); Connector connector = catalog.getConnector(); ExternalDatabase db = mockDb("REMOTE_DB"); @@ -551,6 +553,7 @@ public void testMaterializeLatestNullConnectorDegradesToEmptyPin() { // valid empty pin instead of NPE-ing and aborting the whole metadata query (CI 973411 test_mysql_mtmv // collateral). MUTATION: dropping the null-connector guard in materializeLatest -> NPE -> red. ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); PluginDrivenExternalCatalog droppedCatalog = new TestablePluginCatalog((Connector) null, session); ExternalDatabase db = mockDb("REMOTE_DB"); PluginDrivenMvccExternalTable table = @@ -1363,6 +1366,7 @@ private static Fixture build(List partitions, boolean ti Type partitionColType) { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); ConnectorTableHandle pinnedHandle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog(metadata, session); @@ -1477,6 +1481,11 @@ public ConnectorSession buildConnectorSession() { return session; } + @Override + public ConnectorSession buildCrossStatementSession() { + return buildConnectorSession(); + } + @Override protected List listDatabaseNames() { return Collections.emptyList(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java index e423aeee5645f7..3f7a262680fcf2 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalogDdlRoutingTest.java @@ -37,6 +37,7 @@ import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.ddl.BranchChange; import org.apache.doris.connector.api.ddl.ConnectorColumnPosition; @@ -99,6 +100,7 @@ public void setUp() { connector = Mockito.mock(Connector.class); metadata = Mockito.mock(ConnectorMetadata.class); session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); // Construct with the real Env singleton (the constructor is Env-safe), then @@ -1352,6 +1354,11 @@ public ConnectorSession buildConnectorSession() { return sessionMock; } + @Override + public ConnectorSession buildCrossStatementSession() { + return buildConnectorSession(); + } + @Override public ExternalDatabase getDbNullable(String dbName) { return dbNullableResult; diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabaseTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabaseTest.java index c7dcae846eaa51..d1b3390b553f61 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabaseTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalDatabaseTest.java @@ -21,6 +21,7 @@ import org.apache.doris.connector.api.ConnectorDatabaseMetadata; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.datasource.ExternalCatalog; import org.junit.jupiter.api.Assertions; @@ -45,9 +46,11 @@ private static PluginDrivenExternalCatalog catalogReturning(ConnectorDatabaseMet Connector connector = Mockito.mock(Connector.class); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.getConnector()).thenReturn(connector); Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); return catalog; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTablePartitionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTablePartitionTest.java index 7d3e344ddfed75..d6be2c1bad6acd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTablePartitionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTablePartitionTest.java @@ -27,6 +27,7 @@ import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.handle.ConnectorTableHandle; @@ -126,6 +127,7 @@ public void testGetNameToPartitionItemsBuildsFromConnectorByRemoteNames() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("max_compute", metadata, session); ExternalDatabase db = mockDb("REMOTE_DB"); @@ -155,7 +157,7 @@ public void testGetNameToPartitionItemsEmptyWhenNotPartitioned() { Collections.emptyList(), Collections.emptyList()); ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); TestablePluginCatalog catalog = new TestablePluginCatalog( - "max_compute", metadata, Mockito.mock(ConnectorSession.class)); + "max_compute", metadata, noneScopedSession()); PluginDrivenExternalTable table = tableWithCacheValue( cacheValue, catalog, mockDb("REMOTE_DB"), "REMOTE_TBL"); @@ -169,6 +171,7 @@ public void testGetNameToPartitionItemsEmptyWhenNotPartitioned() { public void testInitSchemaExtractsPartitionColumnsMappingRemoteNames() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("max_compute", metadata, session); ExternalDatabase db = mockDb("REMOTE_DB"); @@ -209,6 +212,7 @@ public void testInitSchemaExtractsPartitionColumnsMappingRemoteNames() { public void testInitSchemaNoPartitionsWhenPropAbsent() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("max_compute", metadata, session); Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) @@ -274,6 +278,14 @@ public void testGetTablePropertiesEmptyWhenConnectorEmitsNone() { // ==================== helpers ==================== + /** A ConnectorSession mock that reports the off-context NONE statement scope (matches the real + * interface default), so the PluginDrivenMetadata funnel does not NPE on getStatementScope(). */ + private static ConnectorSession noneScopedSession() { + ConnectorSession s = Mockito.mock(ConnectorSession.class); + Mockito.when(s.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); + return s; + } + private static ConnectorPartitionInfo partition(String name, String year, String month) { Map values = new LinkedHashMap<>(); values.put("YEAR", year); @@ -305,7 +317,7 @@ private static List columnNames(List columns) { private static PluginDrivenExternalTable tableWithCacheValue(SchemaCacheValue cacheValue) { return tableWithCacheValue(cacheValue, new TestablePluginCatalog("max_compute", Mockito.mock(ConnectorMetadata.class), - Mockito.mock(ConnectorSession.class)), + noneScopedSession()), mockDb("REMOTE_DB"), "REMOTE_TBL"); } @@ -376,6 +388,11 @@ public ConnectorSession buildConnectorSession() { return session; } + @Override + public ConnectorSession buildCrossStatementSession() { + return buildConnectorSession(); + } + @Override protected List listDatabaseNames() { return Collections.emptyList(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTableRowCountTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTableRowCountTest.java index 36fead68e1b13a..f9e6e234fc98bd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTableRowCountTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTableRowCountTest.java @@ -23,6 +23,7 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorTableStatistics; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.datasource.ExternalDatabase; @@ -223,6 +224,7 @@ private static PluginDrivenExternalTable tableWith(Optional schema, List partitionColumnIndexes) { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) .thenReturn(Optional.of(handle)); @@ -264,6 +266,7 @@ private static PluginDrivenExternalTable tableReturning( Optional stats, List schema) { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); if (stats == null) { Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) .thenReturn(Optional.empty()); @@ -325,6 +328,11 @@ public ConnectorSession buildConnectorSession() { return session; } + @Override + public ConnectorSession buildCrossStatementSession() { + return buildConnectorSession(); + } + @Override protected List listDatabaseNames() { return Collections.emptyList(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTableTest.java index 62d93b6c46f620..c08cf81b13472a 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTableTest.java @@ -25,6 +25,7 @@ import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.ConnectorViewDefinition; @@ -85,6 +86,17 @@ public void clearCtx() { ConnectContext.remove(); } + /** + * A ConnectorSession mock that reports the off-context {@link ConnectorStatementScope#NONE} scope, so the + * {@code PluginDrivenMetadata.get(session, connector)} funnel runs {@code connector.getMetadata(session)} on + * every call instead of NPE-ing on a null scope (a plain mock's getStatementScope() would return null). + */ + private static ConnectorSession noneScopedSession() { + ConnectorSession s = Mockito.mock(ConnectorSession.class); + Mockito.when(s.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); + return s; + } + // ==================== §4.4 W3: per-handle write-admission capability probes ==================== /** @@ -105,7 +117,9 @@ private static PluginDrivenExternalTable capabilityTable(boolean handlePresent, Mockito.when(connector.requiresMaterializeStaticPartitionValues(Mockito.any())).thenReturn(true); PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.getConnector()).thenReturn(connector); - Mockito.when(catalog.buildConnectorSession()).thenReturn(Mockito.mock(ConnectorSession.class)); + ConnectorSession session = noneScopedSession(); + Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class, Mockito.CALLS_REAL_METHODS); Deencapsulation.setField(table, "catalog", catalog); return table; @@ -164,7 +178,9 @@ private static PluginDrivenExternalTable writeTargetTable(ConnectorTableHandle r Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.getConnector()).thenReturn(connector); - Mockito.when(catalog.buildConnectorSession()).thenReturn(Mockito.mock(ConnectorSession.class)); + ConnectorSession session = noneScopedSession(); + Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class, Mockito.CALLS_REAL_METHODS); Deencapsulation.setField(table, "catalog", catalog); return table; @@ -204,7 +220,9 @@ private static PluginDrivenExternalTable syntheticPredicateTable(ConnectorTableH Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.getConnector()).thenReturn(connector); - Mockito.when(catalog.buildConnectorSession()).thenReturn(Mockito.mock(ConnectorSession.class)); + ConnectorSession session = noneScopedSession(); + Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class, Mockito.CALLS_REAL_METHODS); Deencapsulation.setField(table, "catalog", catalog); return table; @@ -260,6 +278,7 @@ private static PluginDrivenExternalTable pluginTable(List synth Mockito.when(metadata.getTableHandle(Mockito.any(), Mockito.any(), Mockito.any())) .thenReturn(handlePresent ? Optional.of(handle) : Optional.empty()); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Connector connector = Mockito.mock(Connector.class); Mockito.when(connector.getWritePlanProvider()).thenReturn(writeProviderPresent ? provider : null); // Production now selects the write provider per-handle; a plain Mockito mock does not run the interface @@ -270,6 +289,7 @@ private static PluginDrivenExternalTable pluginTable(List synth PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.getConnector()).thenReturn(connector); Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class, Mockito.CALLS_REAL_METHODS); @@ -619,6 +639,7 @@ private static PluginDrivenExternalTable pluginViewTable(Set inv.getArgument(3)); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Connector connector = Mockito.mock(Connector.class); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); ExternalDatabase db = Mockito.mock(ExternalDatabase.class); @@ -825,6 +854,7 @@ public void initSchemaUsesTableHandlePathForNonView() { PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.getConnector()).thenReturn(connector); Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class, Mockito.CALLS_REAL_METHODS); Deencapsulation.setField(table, "catalog", catalog); @@ -851,6 +881,7 @@ public void systemTableOverridesResolveIsViewToFalse() { Mockito.when(metadata.viewExists(Mockito.any(), Mockito.anyString(), Mockito.anyString())) .thenReturn(true); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Connector connector = Mockito.mock(Connector.class); Mockito.when(connector.getCapabilities()).thenReturn(EnumSet.of(ConnectorCapability.SUPPORTS_VIEW)); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); @@ -859,6 +890,7 @@ public void systemTableOverridesResolveIsViewToFalse() { PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.getConnector()).thenReturn(connector); Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); PluginDrivenSysExternalTable sys = Mockito.mock(PluginDrivenSysExternalTable.class, Mockito.CALLS_REAL_METHODS); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeSysHandleTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeSysHandleTest.java index 432f2a372d8d51..333e7229b8db7d 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeSysHandleTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeSysHandleTest.java @@ -23,6 +23,7 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.datasource.ExternalDatabase; import org.apache.doris.datasource.SessionContext; @@ -69,6 +70,7 @@ public class PluginDrivenScanNodeSysHandleTest { public void createThreadsSysHandleNotBaseHandleForSysTable() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); // Two DISTINCT handles: a base-table handle and the connector's system-table handle. // The base handle stands in for the NORMAL PaimonTableHandle (forceJni=false) that raw // getTableHandle would yield; the sys handle stands in for the force-JNI sys handle. @@ -121,6 +123,7 @@ public void createUsesBaseHandleForNormalTableUnchanged() { // fix is behavior-preserving for normal tables (max_compute/jdbc/etc.). ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); ConnectorTableHandle baseHandle = Mockito.mock(ConnectorTableHandle.class); TestablePluginCatalog catalog = new TestablePluginCatalog("paimon", metadata, session); @@ -198,6 +201,11 @@ public ConnectorSession buildConnectorSession() { return session; } + @Override + public ConnectorSession buildCrossStatementSession() { + return buildConnectorSession(); + } + @Override protected List listDatabaseNames() { return Collections.emptyList(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommandPluginDrivenTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommandPluginDrivenTest.java index 1463023182a0ee..3bfb588129443c 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommandPluginDrivenTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowPartitionsCommandPluginDrivenTest.java @@ -25,6 +25,7 @@ import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.datasource.ExternalDatabase; import org.apache.doris.datasource.ExternalTable; @@ -66,6 +67,7 @@ public void testHandlerRoutesToSpiWithRemoteNames() throws Exception { ShowPartitionsCommand command = new ShowPartitionsCommand(tableName, null, null, -1L, -1L, false); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Connector connector = Mockito.mock(Connector.class); ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); @@ -81,6 +83,7 @@ public void testHandlerRoutesToSpiWithRemoteNames() throws Exception { Mockito.doReturn(db).when(catalog).getDbOrAnalysisException("db"); Mockito.doReturn(table).when(db).getTableOrAnalysisException("t"); Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); Mockito.when(catalog.getConnector()).thenReturn(connector); Mockito.when(connector.getMetadata(session)).thenReturn(metadata); Mockito.when(metadata.getTableHandle(session, "remote_db", "remote_tbl")) @@ -111,6 +114,7 @@ public void testHandlerEmitsFiveColumnsWhenConnectorDeclaresPartitionStats() thr ShowPartitionsCommand command = new ShowPartitionsCommand(tableName, null, null, -1L, -1L, false); ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Connector connector = Mockito.mock(Connector.class); ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); @@ -128,6 +132,7 @@ public void testHandlerEmitsFiveColumnsWhenConnectorDeclaresPartitionStats() thr Mockito.doReturn(db).when(catalog).getDbOrAnalysisException("db"); Mockito.doReturn(table).when(db).getTableOrAnalysisException("t"); Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); Mockito.when(catalog.getConnector()).thenReturn(connector); Mockito.when(connector.getMetadata(session)).thenReturn(metadata); // The capability that flips SHOW PARTITIONS to the 5-column rich result (D-045). diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java index 51b4495b2df54d..0111ef24649008 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java @@ -31,6 +31,7 @@ import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; @@ -496,6 +497,9 @@ private static final class Fixture { @SuppressWarnings("unchecked") Fixture() { props.put("snapshot_id", "200"); + // The funnel PluginDrivenMetadata.get(session, connector) reads session.getStatementScope(); a bare + // Mockito mock returns null (NPE), so pin the interface-default NONE scope (runs getMetadata per call). + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); Mockito.when(connector.getProcedureOps()).thenReturn(procedureOps); // execute() selects the ops per-handle (getProcedureOps(handle)); a Mockito mock does NOT run the @@ -550,6 +554,11 @@ public ConnectorSession buildConnectorSession() { return session; } + @Override + public ConnectorSession buildCrossStatementSession() { + return buildConnectorSession(); + } + @Override protected List listDatabaseNames() { return Collections.emptyList(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/MetadataGeneratorPluginDrivenTest.java b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/MetadataGeneratorPluginDrivenTest.java index 6bebc9a5f87884..2af2683bccc5fe 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/MetadataGeneratorPluginDrivenTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/MetadataGeneratorPluginDrivenTest.java @@ -20,6 +20,7 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; @@ -61,12 +62,14 @@ private TFetchSchemaTableDataResult invokeDeal(PluginDrivenExternalCatalog catal @Test public void testRoutesToSpiWithRemoteNamesAndBuildsRows() throws Exception { ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Connector connector = Mockito.mock(Connector.class); ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); Mockito.when(catalog.getConnector()).thenReturn(connector); Mockito.when(connector.getMetadata(session)).thenReturn(metadata); @@ -93,11 +96,13 @@ public void testRoutesToSpiWithRemoteNamesAndBuildsRows() throws Exception { @Test public void testAbsentHandleYieldsEmptyOkResult() throws Exception { ConnectorSession session = Mockito.mock(ConnectorSession.class); + Mockito.when(session.getStatementScope()).thenReturn(ConnectorStatementScope.NONE); Connector connector = Mockito.mock(Connector.class); ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.buildConnectorSession()).thenReturn(session); + Mockito.when(catalog.buildCrossStatementSession()).thenReturn(session); Mockito.when(catalog.getConnector()).thenReturn(connector); Mockito.when(connector.getMetadata(session)).thenReturn(metadata); From cc2fc95e5fbe24a7af64c28fda313e9b3c184a43 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 19:37:19 +0800 Subject: [PATCH 33/47] [doc](catalog) per-statement metadata: C3 read-side rerouting checklist + progress/handoff Adds the verified seam-by-seam checklist for the read-side rerouting step (66 connector-metadata factory seams partitioned into 51 reroute / 7-9 force-NONE loaders / 8 write-deferred, cross-checked against current line numbers by an adversarial recon), records the user's two decisions (name-mapping seams -> force-NONE; loaders route through the funnel with a NONE session for a zero-exception anti-drift gate), and updates progress / tasklist / handoff to point the next session at the anti-drift gate that closes the read keystone and then the HMS heterogeneous-gateway sibling fan-out. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../per-statement-table-owner-port/HANDOFF.md | 64 +++++------ .../C3-read-reroute-verified-checklist.md | 100 ++++++++++++++++++ .../progress.md | 10 ++ .../tasklist.md | 2 +- 4 files changed, 145 insertions(+), 31 deletions(-) create mode 100644 plan-doc/per-statement-table-owner-port/designs/C3-read-reroute-verified-checklist.md diff --git a/plan-doc/per-statement-table-owner-port/HANDOFF.md b/plan-doc/per-statement-table-owner-port/HANDOFF.md index 06468b43f64919..dd85d3e2e41a60 100644 --- a/plan-doc/per-statement-table-owner-port/HANDOFF.md +++ b/plan-doc/per-statement-table-owner-port/HANDOFF.md @@ -5,54 +5,58 @@ --- -# 🆕 本轮（2026-07-19 session 3）已完成：**定稿分期 + 落地读取键石的 C1 地基 + C2 关闭接线** +# 🆕 本轮（2026-07-19 session 4）已完成：**落地读取侧改道 + 后台读穿纠正（读取键石主体收官）** ## 一句话结果 -- 用户拍板扩展范围并定**分期推进**：读写共用、缓存隔离都做，但排序 + 独立提交；后台"传句柄到后台线程"被取证证伪 → 改"读穿 + 修隐患"；缓存"list ≠ load"已确认 → 缓存隔离是**真实安全修复**。定稿见 `designs/expanded-scope-phasing-and-security-decisions.md`。 -- **已实现并提交（均编译通过 + checkstyle 0 + 单测全绿）**： - - **C1 地基** `5b7312f9d1f`：SPI 加 `getOrCreateMetadata`/`closeAll` 默认方法 + `ConnectorStatementScopeImpl.closeAll`(幂等) + 静态漏斗 `PluginDrivenMetadata.get`。字节中性（无生产路径接进漏斗）。 - - **C2 关闭接线** `12f3e95239b`：**两层关闭**（主=getSplits 注册 `scope::closeAll`；兜底=`StatementContext.close` 守卫 closeAll + `proxyExecute` finally；reset closeAll-before-null；retry 每次重置）。P1 关闭仍 no-op，行为中性。 -- 取证 workflow：`wf_8b907b93-e9f`（扩展范围三块）、`wf_9250330b-e81`（关闭接线生命周期，**推翻了原"scope 创建处注册"会泄漏**）。 +- 动手前先过对抗式核对关（workflow `wf_6e2967a9-1a2`），把改道表逐条对当前代码行号重核 → `designs/C3-read-reroute-verified-checklist.md`：fe-core 连接器 `getMetadata` 工厂缝**共 66 处**完整切四类（相加=66、零遗漏/重叠/未分类）。两处偏差经用户拍板。 +- **已实现（全绿，未提交）**： + - **助手** `PluginDrivenExternalCatalog.buildCrossStatementSession()`：镜像 `buildConnectorSession`（保留凭证），只多强制 `withStatementScope(NONE)`。 + - **9 处后台读穿**（含名字映射两缝 `fromRemoteDatabaseName`/`fromRemoteTableName` 经拍板归入）：改用助手 + 走统一入口 `PluginDrivenMetadata.get`；**含 `fetchRowCount` ANALYZE 隐患修复**（不必单改统计模块）。源码证实「NONE 走漏斗」与裸直调**字节等价**。 + - **49 处改道**：读/DDL/命令/多版本从 `connector.getMetadata(session)` 改调 `PluginDrivenMetadata.get(session, connector)`。 + - **扫描节点存字段**：`volatile cachedMetadata` + `metadata()`；8 处 per-method 改调它，静态 `create` 直调入口。 + - **测试适配**（workflow `wf_fbb60841-365`，11 agents 并行）：补 `getStatementScope()→NONE` stub + 测试替身补 `buildCrossStatementSession` override；新守门 `ConnectorSessionImplTest.explicitNoneStatementScopeWinsOverLiveContext`。 +- **验证**：目标测试 247 全绿（先红 130 error+30 fail）+ ConnectorSessionImplTest 18 绿 + fe-core checkstyle 0 违规 + 主编译 SUCCESS。 --- -# ➡️ 下一个 session = **STEP 1 的 C3：读取侧改道 + 后台读穿纠正（本阶段最大一步，动手前先过清单 + 配对抗复核）** +# ➡️ 下一个 session = **读取键石收官（防漂移门禁）→ 然后 HMS 异构网关兄弟扇出** ## 第一件事（先读） -1. 读 `designs/expanded-scope-phasing-and-security-decisions.md`（分期定稿 + §6 C1/C2 进展 + 关闭接线残留风险）。 -2. 读 `designs/P1-implementation-design.md`：§2 的 66 缝改道表（C3 的清单）；§3 扫描节点存字段；**§4 已更正为两层关闭**（C2 已实现，勿再按旧"scope 创建处注册"）。 -3. 读架构记忆 `iceberg-table-resolution-cache-scoping`。 - -## C3 要做的（读取侧改道 + 后台纠正） -- **读/扫描/DDL/MVCC/misc 改道**（~55 处 on-thread）：把 `connector.getMetadata(session)` 改调 `PluginDrivenMetadata.get(session, connector)`。清单见 §2 组 A/B/C/E/misc。 -- **扫描节点存字段**（组 C）：`PluginDrivenScanNode` 加懒字段 `cachedMetadata` + `metadata()`，8 处 per-method 重取改调它（`volatile` 稳态）。 -- **后台读穿纠正**（用户拍板）：7 处跨语句 loader（`listDatabaseNames`/`listTableNamesFromRemote`/`initSchema`/`getColumnStatistic`/`getChunkSizes`/`fetchRowCount`/`MetadataGenerator.dealPluginDrivenCatalog`）**显式建 NONE session 读穿**（`ConnectorSessionBuilder.withStatementScope(NONE)` 或新增 `buildCrossStatementSession()`）；**修 `fetchRowCount` 在 ANALYZE 线程（`AnalysisManager.buildAnalysisJobInfo`）被错误绑定到语句作用域的隐患**。 -- **⚠ 别误改道 `PluginDrivenExternalTable.resolveWriteTargetHandle`**（藏在读文件里的**写专用**点）——标记为写 seam，留 STEP 3。 -- 守门：一语句一表加载计数=1（对照 NONE=N）；跨 catalog/queryId 隔离。 -- **动手前**：先把 §2 改道表逐条核对当前代码行号（可能漂移），列清单给用户过一遍，并计划一轮对抗式复核（clean-room 偏好）。 - -## 后续步（STEP 2/3/4，详见分期定稿 §4） -- **STEP 2 HMS 兄弟**：键含属主 label + 兄弟 getMetadata **及 beginTransaction** 进漏斗 + 异构网关 e2e。 -- **STEP 3 写入共用**：3a 无状态写点改道（含被标记的 `resolveWriteTargetHandle`）；3b `ConnectorTransaction` 归属上移 `CatalogStatementTransaction`。闸门=读写身份等价 + 保留 hive 起写刷新 + 保留 tx↔session 绑定。纠缠点：iceberg 起写复用表读 `IcebergStatementScope.sharedTable`（P2 计划删）——3b 时定"先接旧 vs 先做 iceberg 最小 P2 前置"。 +1. 读 `designs/C3-read-reroute-verified-checklist.md`（已实施；§1 四类分区、§2 强制 NONE 机制、用户两处拍板）。 +2. 读 `designs/P1-implementation-design.md` §5（HMS sibling 扇出蓝图：三连接器共享 catalogId → key 加属主 label）+ §8（arch 门禁蓝图）。 +3. 读架构记忆 `iceberg-table-resolution-cache-scoping`、`catalog-spi-hms-hiveversionutil-gate-false-positive`（门禁误报先例）。 + +## A. 收官：防漂移门禁（读取键石最后一步，小） +- 目标：加一道构建期门禁（bash grep，仿 `tools/check-connector-imports.sh` 风格 + 自测），确保 fe-core 里裸 `connector.getMetadata(` **只允许出现在** `PluginDrivenMetadata.java`（漏斗自身的 factory lambda）。 +- **本轮已把读穿 loader 也走统一入口 → 门禁零例外**（无需白名单）。当前 fe-core 里合法裸调用只剩：`PluginDrivenMetadata.java:54`（漏斗 lambda）+ 写路径 8 处（`PhysicalPlanTranslator`×2 / `BindSink`×2 / `IcebergRowLevelDmlTransform` / `PhysicalIcebergMergeSink` / `PluginDrivenInsertExecutor` / `PluginDrivenExternalTable.resolveWriteTargetHandle:133`）。 +- **决策点（交用户）**：写路径 8 处现仍裸直调（留写入共用步骤）。门禁要么(a)现在只锁"非写文件"、写文件暂豁免（写入共用步骤再收）；要么(b)推迟整个门禁到写入共用步骤后一次落。建议(a)：现在就锁读侧防漂移，写文件显式列为"暂待写入步骤"豁免。落点见 P1-design §8（checkstyle Regexp 或 fe-connector pom 的独立 exec）。 + +## B. 下一大步：HMS 异构网关兄弟扇出（STEP 2 / P1-design §5） +- 网关已 LIVE（hms 在 `SPI_READY_TYPES`）。三连接器（hive/iceberg/hudi）**共享一个 catalogId** → 若统一入口 key 只用 catalogId 会把三者塌成一个 metadata、派错。 +- **修**：①入口 key 加属主 label（`"metadata:"+catalogId+":"+owner`，owner∈{hive,iceberg,hudi}，在 `resolveSiblingOwner` 命中臂后取 label）；②**兄弟 getMetadata 也进入口**（`siblingMetadata`/`icebergSiblingMetadata`/`hudiSiblingMetadata`）；③只存/返回 `ConnectorMetadata` 接口不 cast；④补异构网关 e2e（`external_table_p2/refactor_catalog_param`，对齐 `hms-iceberg-delegation-needs-e2e`）。 +- 注意：sibling 扇出在**连接器内部**（fe-connector-hive），fe-core arch 门禁看不见 → 由连接器侧单测守（仿 `HiveConnectorSiblingTest`）。 + +## 后续步（详见分期定稿 §4） +- **STEP 3 写入共用**：3a 无状态写点（含 `resolveWriteTargetHandle`）改道进入口；3b `ConnectorTransaction` 归属上移 `CatalogStatementTransaction`。闸门=读写身份等价 + 保留 hive 起写刷新 + 保留 tx↔session 绑定。纠缠点：iceberg 起写复用 `IcebergStatementScope.sharedTable`（P2 计划删）→ 3b 时定先后。 - **STEP 4 缓存隔离（安全）**：`getIdentityShardKey()` SPI → iceberg 三投影缓存分片 + fe-core 表结构缓存 bypass(先)/分片(后) + 防漂移门禁；随 STEP 2 属主键覆盖异构网关；越权 e2e + 威胁模型签字。 ## 关闭接线残留风险（carry-forward，详见分期定稿 §6） -1. 取消/超时非硬栅栏（`SplitAssignment.stop` 只置标志不 join pump）——既有共担，P1 no-op 下无实害，**关闭做实事前须硬化**。 +1. 取消/超时非硬栅栏（`SplitAssignment.stop` 只置标志）——既有共担，P1 no-op 关闭下无实害，**关闭做实事前须硬化**。 2. arrow-flight 异常断连 → 注册表条目留存（无 TTL）——既有共担。 -3. **待确认**：走协调器但从不走 getSplits 的外部目录查询（纯 information_schema / 某元数据 TVF）可能漏关——STEP 3/后续确认。 -4. **🔴 TCCL 自钉扎（硬前置）**：一旦连接器 `close()` 做实事（P2+），主关闭回调 + 兜底两处都须把 TCCL 钉到插件 classloader（连接器 `close()` 自钉扎首选）。 +3. **待确认**：走协调器但从不走 getSplits 的纯 information_schema / 某元数据 TVF 可能漏关。 +4. **🔴 TCCL 自钉扎（硬前置）**：连接器 `close()` 做实事（P2+）后，主关闭回调 + 兜底两处都须把 TCCL 钉到插件 classloader（连接器 `close()` 自钉扎首选）。 ## 铁律 / 闸门提醒 -- 用户 2026-07-19 已豁免铁律 A（fe-core 只减不增）——本重构的 fe-core 净增已获授权。仍守：连接器 connector-agnostic（作用域值 Object）；作用域跨用户即泄漏（STEP 4 威胁模型）。 +- 用户 2026-07-19 已豁免铁律 A（fe-core 只减不增）。仍守：连接器 connector-agnostic（作用域值 Object）；作用域跨用户即泄漏（STEP 4 威胁模型）。 - **动码前探测并发活动**（git log/status + maven 进程 + 近 90s mtime），发现活跃即停手（`concurrent-sessions-shared-worktree-hazard`）。 --- # 🗂 遗留 / 关联 - 分期定稿（现行主线）：`designs/expanded-scope-phasing-and-security-decisions.md`（含 §6 进展 + 残留风险）。 -- STEP 1 蓝图：`designs/P1-implementation-design.md`（§2 改道表；§4 已更正为两层关闭）。 +- STEP 1 蓝图：`designs/P1-implementation-design.md`（§2 改道表；§4 两层关闭；§5 HMS sibling；§8 arch 门禁）。 +- C3 核对清单（已实施）：`designs/C3-read-reroute-verified-checklist.md`。 - 目标架构全景：`designs/trino-parity-metadata-redesign-design.md`。 -- 蓝本 iceberg：`../perf-hotpath-iceberg/`（PERF-07）；代码范本 `IcebergStatementScope`(commit `ea7fd1f6e7a`)+地基 `97bdcd6bdbe`。 +- 本轮 workflow：`wf_6e2967a9-1a2`（C3 逐缝核对+对抗）、`wf_fbb60841-365`（测试适配 11 文件）。前轮 `wf_8b907b93-e9f`/`wf_9250330b-e81`。 - 架构记忆：`iceberg-table-resolution-cache-scoping`。 -- 本轮 workflow journal：`wf_8b907b93-e9f`（扩展范围）、`wf_9250330b-e81`（关闭接线生命周期）；前轮 `wf_72d1e505-75c`/`wf_62cc379e-c6e`/`wf_7e537094-44f`。 -- e2e 一律留连接器进 `SPI_READY_TYPES` 切换阶段统一补；异构网关 e2e 随 STEP 2/4（对齐 `hms-iceberg-delegation-needs-e2e`）。 +- e2e 一律留连接器进 `SPI_READY_TYPES` 切换阶段统一补；异构网关 e2e 随 STEP 2/4。 diff --git a/plan-doc/per-statement-table-owner-port/designs/C3-read-reroute-verified-checklist.md b/plan-doc/per-statement-table-owner-port/designs/C3-read-reroute-verified-checklist.md new file mode 100644 index 00000000000000..c52919558bdb74 --- /dev/null +++ b/plan-doc/per-statement-table-owner-port/designs/C3-read-reroute-verified-checklist.md @@ -0,0 +1,100 @@ +# C3 读取侧改道 —— 逐缝核对清单（动手前，行号已按当前代码校准） + +> 承接 `P1-implementation-design.md` §2/§3 与 `expanded-scope-phasing-and-security-decisions.md` §2。 +> 本文是 C3（读取侧改道 + 后台读穿纠正）**动手前的核对关**产物：把设计里的改道表逐条对当前代码行号重新核实（会漂移），并经一轮对抗式复核（workflow `wf_6e2967a9-1a2`，10 agents：7 census reader + 3 adversarial verifier）。**本文不动代码。** + +--- + +## ✅ 实施状态（2026-07-19 session 4） + +已全部落地并验证：`buildCrossStatementSession` 助手 + 9 处 NONE 读穿（含名字映射两缝，含 fetchRowCount ANALYZE 修复）+ 49 处改道 + 扫描节点 `cachedMetadata`/`metadata()`。测试适配 11 文件 + 新守门 `ConnectorSessionImplTest.explicitNoneStatementScopeWinsOverLiveContext`。验证：目标 247 测全绿 + checkstyle 0 + 主编译 SUCCESS。**剩防漂移门禁**（读取键石收官，见 HANDOFF）。 + +## 0. 结论一句话 + +- fe-core 里连接器 `getMetadata(` 工厂缝 = **66 处**（80 raw grep 减 9 处测试静态 + 4 处漏斗自身 + 1 处 `RuntimeProfile.node.getMetadata()`）。**完整切成四类，相加正好 66，无遗漏/无重叠/无未分类缝。** +- 全部设计命名的行号**零漂移**（例外：扫描节点 9 处 + 尾部两处漂 +13，纯位置漂移，方法未变）。 +- 揪出两处需拍板的偏差（§4）。 + +## 1. 四类分区（66 = 49/51 改道 + 7/9 读穿 + 8 写延后 + 4 漏斗自身在 66 之外） + +> 注：4 处漏斗自身（`PluginDrivenMetadata` 内 3 javadoc + 1 factory lambda）不计入 66。 + +### A. 改道进漏斗（`connector.getMetadata(session)` → `PluginDrivenMetadata.get(session, connector)`） + +| 文件 | 缝 | 当前行 | 线程 | +|---|---|---|---| +| PluginDrivenExternalCatalog（DDL 19 + tableExist 1 = 20 处） | createTable/createDb/dropDb/dropTable/renameTable/truncateTable/addColumn(s)/dropColumn/renameColumn/modifyColumn/reorderColumns/createOrReplaceBranch/createOrReplaceTag/dropBranch/dropTag/addPartitionField/dropPartitionField/replacePartitionField + tableExist | 332,422,501,550,598,670,714,807,823,838,853,868,884,915,931,947,963,988,1003,1018 | on-thread | +| PluginDrivenExternalTable（读 12 处） | getSyntheticScanPredicates(159)/resolveWriteCapabilityHandle(189,读——探写能力但读元数据)/resolveIsView(576)/getViewText(594)/getShowCreateTableDdl(617)/fetchSyntheticWriteColumns(727,读——getFullSchema 路径)/getNameToPartitionItems(820)/getNameToPartitionValues(881)/getMetadataTableRows(918)/getComment(950)/getSupportedSysTables(981)/（第 12 处 1304） | 159,189,576,594,617,727,820,881,918,950,981,1304 | on-thread/planning | +| PluginDrivenExternalDatabase.getLocation | 1 | 84 | on-thread | +| PluginDrivenScanNode（扫描 9 处，**存字段而非裸改**） | create(静态工厂,206) + convertPredicate(806)/tryPushDownLimit(846)/tryPushDownProjection(865)/pinMvccSnapshot(910)/pinRewriteFileScope(1055)/pinTopnLazyMaterialize(1074)/buildColumnHandles(1902)/buildRemainingFilter(1939) | 见左 | planning-thread | +| PluginDrivenMvccExternalTable（3 处） | materializeLatest(143,mixed)/loadSnapshot(358,on-thread)/resolveFreshnessProbe(725,background) | 143,358,725 | 见左 | +| misc（4 处命令/TVF） | ShowPartitionsCommand(285)/ConnectorExecuteAction(128)/CallExecuteStmtFunc(101)/JdbcQueryTableValueFunction(52) | 见左 | on-thread | + +**扫描节点存字段细节**：加懒字段 `private volatile ConnectorMetadata cachedMetadata;` + `metadata()` 访问器（`if(cachedMetadata==null) cachedMetadata = PluginDrivenMetadata.get(connectorSession, connector);`），8 处 per-method 全改调 `metadata()`。**create(206) 是静态工厂无实例** → 直接调 `PluginDrivenMetadata.get(session, connector)`（不能用实例访问器）。字段现状：`connector:148`、`connectorSession:149`、`currentHandle:152`；镜像现有 `volatile resolvedScanProvider:184`。8 处均在单规划线程（并发 appendBatch 之前）。 + +### B. 后台读穿加载器（跨语句缓存 loader，**强制 NONE 会话**，绝不绑定语句作用域） + +| 文件 | 缝 | 当前行 | 线程 | +|---|---|---|---| +| PluginDrivenExternalCatalog.listDatabaseNames | 1 | 295 | background | +| PluginDrivenExternalCatalog.listTableNamesFromRemote | 1 | 311 | background | +| PluginDrivenExternalTable.initSchema | 1 | 460 | background(schema-cache) | +| PluginDrivenExternalTable.getColumnStatistic | 1 | 1052 | background(stats-cache) | +| PluginDrivenExternalTable.getChunkSizes | 1 | 1087 | analyze-thread | +| PluginDrivenExternalTable.fetchRowCount | 1 | 1153 | **mixed**（含 ANALYZE 执行线程，见 §3） | +| MetadataGenerator.dealPluginDrivenCatalog | 1 | 1329 | background(BE-driven) | + +### C. 写入路径（**留后续步骤，C3 不碰**，共 8 处） + +`resolveWriteTargetHandle(PluginDrivenExternalTable:133)` + PhysicalPlanTranslator(612,660) + PluginDrivenInsertExecutor(206) + BindSink(673,712) + IcebergRowLevelDmlTransform(112) + PhysicalIcebergMergeSink(303)。 + +--- + +## 2. 强制 NONE 的机制（已源码证实为「与裸直调字节等价」） + +- `ConnectorSessionBuilder.withStatementScope(ConnectorStatementScope.NONE)` 显式短路 `captureStatementScope()` 的第一道 `!= null` 守卫 → 与线程 ConnectContext 状态**完全无关地强制 NONE**（`ConnectorSessionBuilder.java:190-203`）。 +- `ConnectorStatementScope.NONE.computeIfAbsent(key,loader) = loader.get()`（每次跑工厂、零缓存）；`getOrCreateMetadata` 默认走它；`closeAll` 默认 no-op → NONE 无可关之物。 +- 故 `PluginDrivenMetadata.get(NONE会话, connector)` = **每次新建、零留存**，SPI 文档原话「byte-identical to building metadata every time」。 +- 7 处 loader 现状全是 `.buildConnectorSession()`（读 ConnectContext，有 ctx 即绑活作用域=泄漏隐患）。**修法 = 新增 `PluginDrivenExternalCatalog.buildCrossStatementSession()`**：镜像 `buildConnectorSession()` 两分支，末尾统一 `.withStatementScope(ConnectorStatementScope.NONE)`。7 处改调它。 + +--- + +## 3. ANALYZE 隐患（已证实，修法并入 fetchRowCount 强制 NONE） + +- `AnalyzeTableCommand.doRun` → `AnalysisManager.buildAnalysisJobInfo:349` → `:418` 在 `getRowCount()<=0` 时**同步直调** `table.fetchRowCount()`，全程在 ANALYZE 语句执行线程（ConnectContext + StatementContext 均活）。 +- `fetchRowCount:1152` 经 `buildConnectorSession()` 捕获该活作用域。 +- **今日无害**：fetchRowCount 现走**裸直调** `connector.getMetadata(session)`（不经漏斗），故什么都没 memo 进作用域，会话只是持了个引用。隐患是**前瞻性**的：一旦 fetchRowCount 改走漏斗（且 metadata 挂真实可关资源），会把资源钉在整个 ANALYZE 生命周期、并在语句末误关。 +- **修法**：fetchRowCount 用 `buildCrossStatementSession()` 强制 NONE。**无需单独改 AnalysisManager**——强制 NONE 后无论走不走漏斗都读穿。 + +--- + +## 用户拍板（2026-07-19） + +- **偏差①**：名字映射两缝 `fromRemoteDatabaseName(1106)`/`fromRemoteTableName(1112)` → **归入强制 NONE 读穿**（→ 读穿 9 处、改道 49 处）。 +- **偏差②**：读穿 loader → **走统一入口 + 传 NONE 会话**（甲案，门禁零例外）。 + +## 4. 两处需拍板的偏差（对抗复核揪出） + +### 偏差①：名字映射两缝 `fromRemoteDatabaseName(1106)`/`fromRemoteTableName(1112)` +- 设计原判：on-request 的 DDL 改道（进 A 类改道）。 +- 复核实证（两 agent 独立）：其**全部调用方跑在离请求线程的名字缓存 loader / buildDbForInit / buildTableForInit / metastore-event-sync 路径**（`ExternalCatalog:586,970`；`ExternalDatabase:224,277,659`），与后台 loader 同族。裸改道进漏斗 → 若某次同步首载落在有活 ConnectContext 的请求线程，会把 memo 值绑进**活得比语句久的名字缓存**。 +- **建议**：与其它 7 处 loader 同等对待——**强制 NONE**（→ 读穿 9 处、改道 49 处）。 + +### 偏差②：读穿 loader 的落地形态（漏斗+NONE vs 裸直调+门禁白名单） +- 旧设计 §8 选「保留裸直调 + 防漂移门禁白名单」（当时理由「更清晰、避免塞残留 scope」，该理由在强制 NONE 后已失效——NONE 无 scope 可塞）。 +- 已证实「漏斗+NONE」与「裸直调」**字节等价**，且让防漂移门禁**零例外**（fe-core 里不再有合法的裸 `connector.getMetadata(`）。 +- **建议**：读穿 loader 也走漏斗（传 NONE 会话），门禁更干净。 + +--- + +## 5. 落地子提交建议（C3 内部，每步独立可编译/回退） + +1. 新增 `buildCrossStatementSession()` 助手（连接器无关，强制 NONE）。 +2. 后台读穿：7（或 9）处 loader 改用助手 + 走漏斗；含 fetchRowCount ANALYZE 隐患修复。守门：加载计数=1(对照 NONE=N)、跨 catalog/queryId 隔离、ANALYZE 首次不绑活作用域。 +3. 读/DDL/misc/MVCC 改道：49（或 51）处裸改道进漏斗。 +4. 扫描节点存字段：加 `cachedMetadata`+`metadata()`，9 处改调。 +- 防漂移门禁（bash grep，仿 `check-connector-imports.sh`）择机随 STEP 1 收尾补（形态取决于偏差②）。HMS 兄弟扇出 = 下一大步，不在 C3。 + +## 6. 已核实为「不动」 + +- 写入 8 处（§1.C）；`getMetadataTableRows()` 数据调用（PluginDrivenExternalTable:923，非工厂缝）；`TestExternalCatalog` 9 处测试静态 Map；`RuntimeProfile.node.getMetadata():260`；`PluginDrivenMetadata` 漏斗自身 4 处。 diff --git a/plan-doc/per-statement-table-owner-port/progress.md b/plan-doc/per-statement-table-owner-port/progress.md index 91ffaf040ca11f..24de2745e12ac9 100644 --- a/plan-doc/per-statement-table-owner-port/progress.md +++ b/plan-doc/per-statement-table-owner-port/progress.md @@ -62,3 +62,13 @@ - **C2 关闭接线**（commit `12f3e95239b`）：改为**两层关闭**——主关闭 `PluginDrivenScanNode.getSplits` 注册 `scope::closeAll`（对象捕获、跳 NONE、同 read-txn queryId 键，只对走协调器语句触发、pump 静默后）；兜底 `StatementContext.close()` 的 `isReturnResultFromLocal` 守卫 closeAll（直连 `executeQuery` finally + `proxyExecute` 新增 finally 覆盖转发主节点）；`resetConnectorStatementScope` 改 closeAll-before-null；`handleQueryWithRetry` 每重试重置。核实 `releasePlannerResources` 幂等故转发路径补 close 安全。单测 reset-先关/close-本地关/close-异步延后。P1 关闭仍 no-op，行为中性。编译 + checkstyle 0 + 11 测全绿。 - 更正 `P1-implementation-design.md` §4（旧"scope 创建处注册"→两层关闭）；残留风险（取消非硬栅栏、arrow-flight 断连、待确认的非-getSplits 外部查询、**TCCL 自钉扎硬前置**）记入分期定稿 §6。 - **下一步**：见 HANDOFF —— 下个 session 做 C3（读取侧改道 ~55 缝 + 扫描节点存字段 + 7 处后台读穿纠正 + 修 fetchRowCount 隐患），动手前先核行号列清单 + 配对抗复核。用户将在下个 session 开新任务。 + +## 2026-07-19 — session 4：落地读取侧改道 + 后台读穿纠正（读取键石收官主体） + +- **动手前核对关（对抗复核 workflow `wf_6e2967a9-1a2`，10 agents：7 逐文件普查 + 3 对抗验证）**：把改道表逐条对当前代码行号重核。结论落 `designs/C3-read-reroute-verified-checklist.md`：fe-core 连接器 `getMetadata` 工厂缝**共 66 处**，完整切成四类，相加=66、零遗漏/零重叠/零未分类；设计命名行号零漂移（仅扫描节点尾部 +13 行位置漂移）。对抗复核 CONFIRM 写专用点 `resolveWriteTargetHandle` 唯一生产调用方是插入执行器开事务、读路径够不到，必须留后续。 +- **两处偏差交用户拍板 + 拍定**：①名字映射两缝（`fromRemoteDatabaseName`/`fromRemoteTableName`）经复核跑在离请求线程的缓存加载路径 → 归入强制 NONE 读穿（读穿 9 处、改道 49 处）；②读穿 loader 走统一入口 + 传 NONE 会话（门禁零例外）。 +- **落地（首次改产品代码，未提交前全绿）**：新增 `PluginDrivenExternalCatalog.buildCrossStatementSession()`（镜像 buildConnectorSession + 强制 NONE，保留凭证）；9 处后台 loader 改用它并走统一入口（含 `fetchRowCount` ANALYZE 隐患修复——不必单改统计模块）；49 处读/DDL/命令/多版本改道进统一入口；扫描节点加 `volatile cachedMetadata` + `metadata()` 访问器（静态 create 直调入口）。源码证实「强制 NONE 走统一入口」与裸直调**字节等价**（NONE 每次跑工厂、零留存）。 +- **测试适配（workflow `wf_fbb60841-365`，11 agents 并行修各文件）**：改道后这些路径经统一入口调 `session.getStatementScope()`，Mockito mock 默认返 null → 漏斗 NPE；按 C1 约定（测试给 NONE 作用域）逐文件补 `getStatementScope()→NONE` stub + 给测试替身补 `buildCrossStatementSession` override（不弱化任何断言/verify）。新增守门 `ConnectorSessionImplTest.explicitNoneStatementScopeWinsOverLiveContext`（活 ctx 下显式 NONE 胜出，锁 ANALYZE 隐患修复的机制）。 +- **验证**：目标测试 247 全绿（先红 130 error+30 fail → 修后 0/0）+ ConnectorSessionImplTest 18 绿 + fe-core checkstyle BUILD SUCCESS 零违规 + fe-core 主编译 BUILD SUCCESS。 +- **提交口径**：4 个逻辑子步（助手/读穿/改道/扫描存字段）在 2 个共享文件里交织，且每提交须保持绿（测试适配须随产品改动同提），逐 hunk 拆分需交互式 git（本环境不支持）→ 按 C1/C2 先例=1 个绿代码提交（含产品+测试适配+守门）+ 1 个文档提交，代码提交体内枚举 4 子步。 +- **下一步**：见 HANDOFF —— 读取键石剩防漂移门禁（形态=统一入口零例外）；下一大步 = HMS 异构网关兄弟扇出（键含属主 label + 兄弟 getMetadata/起事务进同一入口 + 异构网关 e2e）。 diff --git a/plan-doc/per-statement-table-owner-port/tasklist.md b/plan-doc/per-statement-table-owner-port/tasklist.md index bf299d7c151baf..b19652d4032a9c 100644 --- a/plan-doc/per-statement-table-owner-port/tasklist.md +++ b/plan-doc/per-statement-table-owner-port/tasklist.md @@ -10,7 +10,7 @@ | ID | 步 | 状态 | 内容 | 依赖 | commit | |---|---|---|---|---|---| -| RD-1 | STEP 1 读取键石 | 🚧 进行中（C1+C2 已提交，C3 待做） | C1 地基 ✅ + C2 关闭接线 ✅ + C3 读/扫描/DDL/MVCC 改道 + 扫描节点存字段 + 7 处后台 loader 显式 NONE 读穿 + 修 fetchRowCount ANALYZE 隐患（待做） | — | C1 `5b7312f9d1f` · C2 `12f3e95239b` | +| RD-1 | STEP 1 读取键石 | 🚧 进行中（C1+C2+C3 已落地，仅剩防漂移门禁） | C1 地基 ✅ + C2 关闭接线 ✅ + C3 读/扫描/DDL/MVCC 改道（49 处）✅ + 扫描节点存字段 ✅ + 后台 loader 显式 NONE 读穿（9 处，含名字映射两缝）✅ + 修 fetchRowCount ANALYZE 隐患 ✅ + `buildCrossStatementSession` 助手 ✅ + 守门测试 ✅（247+18 绿、checkstyle 0）。剩：防漂移门禁（bash grep，形态=统一入口零例外）⏳ | — | C1 `5b7312f9d1f` · C2 `12f3e95239b` · C3 待提交 | | RD-2 | STEP 2 HMS 兄弟扇出 | ⏳ 待启动 | 键=(catalogId, ownerLabel)；兄弟 getMetadata **及 beginTransaction** 进同一 funnel；异构网关 e2e | RD-1 | | | RD-3 | STEP 3 写入共用 | ⏳ 待启动 | 3a 无状态写点改道进 funnel；3b `ConnectorTransaction` 归属上移 `CatalogStatementTransaction` + commit/rollback vs closeAll 顺序。闸门=读写身份等价 + 保留 hive 起写刷新 + 保留 tx↔session 绑定 | RD-1,RD-2 | | | RD-4 | STEP 4 缓存隔离（安全 track） | ⏳ 待启动（**list≠load 已确认=真实越权**） | `getIdentityShardKey()` SPI → iceberg 三投影缓存 Key 分片 + fe-core 表结构缓存 bypass(先)/分片(后) + 防漂移门禁；随 STEP 2 属主键覆盖异构网关；越权 e2e | 威胁模型签字 + RD-2 | | From 7671086cabfba91996fcea774d991f808057c407 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 20:50:27 +0800 Subject: [PATCH 34/47] [feat](catalog) fe-core: arch gate pinning metadata acquisition to the per-statement funnel Close out the read keystone of the per-statement metadata redesign with a build-time gate so the read/scan/DDL/MVCC seams cannot drift back to bare connector.getMetadata(session) calls. tools/check-fecore-metadata-funnel.sh (+ self-test) greps fe-core main sources and fails the build on any Connector#getMetadata(session) call outside the funnel PluginDrivenMetadata.get(...). Kept silent: the funnel file itself; the no-arg getMetadata() (a different method); comment mentions; and write-path seams carrying a `getMetadata-funnel-exempt` marker (on the call line or the line above it). Wired as a validate-phase exec in fe-core/pom.xml, mirroring tools/check-connector-imports.sh. The 8 write-path seams (INSERT/DELETE/MERGE sink, bind, row-level DML, write target handle) still call getMetadata directly and are marked exempt; the marker is removed and the gate auto-tightens onto them when the write-sharing step reroutes writes through the funnel. Verified: self-test PASS; gate green on the current tree and red on the 8 unmarked write sites; fe-core checkstyle 0 violations; `mvn -pl fe-core validate` fires the gate with BUILD SUCCESS. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- fe/fe-core/pom.xml | 29 ++++ .../plugin/PluginDrivenExternalTable.java | 1 + .../translator/PhysicalPlanTranslator.java | 2 + .../nereids/rules/analysis/BindSink.java | 2 + .../commands/IcebergRowLevelDmlTransform.java | 1 + .../insert/PluginDrivenInsertExecutor.java | 1 + .../physical/PhysicalIcebergMergeSink.java | 1 + tools/check-fecore-metadata-funnel.sh | 126 ++++++++++++++ tools/check-fecore-metadata-funnel.test.sh | 160 ++++++++++++++++++ 9 files changed, 323 insertions(+) create mode 100755 tools/check-fecore-metadata-funnel.sh create mode 100755 tools/check-fecore-metadata-funnel.test.sh diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml index 6ef409718e1e7d..3d542c1acbaafe 100644 --- a/fe/fe-core/pom.xml +++ b/fe/fe-core/pom.xml @@ -978,6 +978,35 @@ under the License. + + + org.codehaus.mojo + exec-maven-plugin + 3.1.0 + + + check-fecore-metadata-funnel + validate + + exec + + + ${project.basedir}/../../tools/check-fecore-metadata-funnel.sh + + ${project.basedir} + + + + + diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java index b6dd3373fecc84..f5edcd3278fd1c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java @@ -130,6 +130,7 @@ public Optional resolveConnectorTableHandle( public ConnectorTableHandle resolveWriteTargetHandle() { PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; ConnectorSession session = pluginCatalog.buildConnectorSession(); + // getMetadata-funnel-exempt: write path, rerouted through the funnel in the write-sharing step return resolveConnectorTableHandle(session, pluginCatalog.getConnector().getMetadata(session)) .orElseThrow(() -> new DorisConnectorException( "Cannot resolve the connector table handle for write target " + getName())); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index f548564a50aca8..47598e2f726dd9 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -609,6 +609,7 @@ private PluginDrivenTableSink buildPluginRowLevelDmlSink( Connector connector = catalog.getConnector(); ConnectorSession connSession = catalog.buildConnectorSession(); + // getMetadata-funnel-exempt: write path, rerouted through the funnel in the write-sharing step ConnectorMetadata metadata = connector.getMetadata(connSession); List connectorColumns = sink.getCols().stream() @@ -657,6 +658,7 @@ public PlanFragment visitPhysicalConnectorTableSink( // Get write config from the connector Connector connector = catalog.getConnector(); ConnectorSession connSession = catalog.buildConnectorSession(); + // getMetadata-funnel-exempt: write path, rerouted through the funnel in the write-sharing step ConnectorMetadata metadata = connector.getMetadata(connSession); // Convert sink columns to connector columns for INSERT SQL generation diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java index 1a7d41cd811f85..9b258d8e49fa72 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java @@ -670,6 +670,7 @@ private void checkConnectorStaticPartitions(PluginDrivenExternalTable table, } PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); ConnectorSession session = catalog.buildConnectorSession(); + // getMetadata-funnel-exempt: write path, rerouted through the funnel in the write-sharing step ConnectorMetadata metadata = catalog.getConnector().getMetadata(session); ConnectorTableHandle handle = metadata.getTableHandle( session, table.getRemoteDbName(), table.getRemoteName()) @@ -709,6 +710,7 @@ private void checkConnectorWritePartitionNames(PluginDrivenExternalTable table, } PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); ConnectorSession session = catalog.buildConnectorSession(); + // getMetadata-funnel-exempt: write path, rerouted through the funnel in the write-sharing step ConnectorMetadata metadata = catalog.getConnector().getMetadata(session); ConnectorTableHandle handle = metadata.getTableHandle( session, table.getRemoteDbName(), table.getRemoteName()) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransform.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransform.java index 6d846ae7408458..a7481debe2eff2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransform.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransform.java @@ -109,6 +109,7 @@ public void checkMode(TableIf table, RowLevelDmlOp op) { private static void checkPluginMode(PluginDrivenExternalTable table, RowLevelDmlOp op) { PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); ConnectorSession session = catalog.buildConnectorSession(); + // getMetadata-funnel-exempt: write path, rerouted through the funnel in the write-sharing step ConnectorMetadata metadata = catalog.getConnector().getMetadata(session); ConnectorTableHandle handle = metadata.getTableHandle( session, table.getRemoteDbName(), table.getRemoteName()) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java index 4bd1ebd8f265e6..fa099afc5f2880 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java @@ -203,6 +203,7 @@ private void ensureConnectorSetup() { (PluginDrivenExternalCatalog) ((ExternalTable) table).getCatalog(); Connector connector = catalog.getConnector(); connectorSession = catalog.buildConnectorSession(); + // getMetadata-funnel-exempt: write path, rerouted through the funnel in the write-sharing step writeOps = connector.getMetadata(connectorSession); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java index e6d4cfddda55fc..2bd2f844872d7b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java @@ -300,6 +300,7 @@ private InsertPartitionFieldResult buildInsertPartitionFieldsFromConnector( PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); Connector connector = catalog.getConnector(); ConnectorSession session = catalog.buildConnectorSession(); + // getMetadata-funnel-exempt: write path, rerouted through the funnel in the write-sharing step ConnectorMetadata metadata = connector.getMetadata(session); // Resolve the handle first so the write provider is selected per-table (a heterogeneous gateway routes // iceberg-on-HMS to its sibling by the handle type); both null-degrade checks keep the non-partitioned diff --git a/tools/check-fecore-metadata-funnel.sh b/tools/check-fecore-metadata-funnel.sh new file mode 100755 index 00000000000000..4f17977b670a01 --- /dev/null +++ b/tools/check-fecore-metadata-funnel.sh @@ -0,0 +1,126 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Arch gate: the per-statement ConnectorMetadata funnel. +# +# Invariant: within a statement, fe-core must reuse exactly ONE ConnectorMetadata +# instance per catalog. Every read / scan / DDL / MVCC seam therefore acquires its +# metadata through the engine-side funnel +# org.apache.doris.datasource.plugin.PluginDrivenMetadata.get(session, connector) +# which memoizes connector.getMetadata(session) on the statement's ConnectorStatementScope +# and closes it deterministically at statement end. A bare connector.getMetadata(session) +# anywhere else in fe-core silently mints a second, unmanaged instance and breaks the +# invariant with no compile error and no test failure — this gate fails the build on it. +# +# Allowed (kept silent): +# 1. PluginDrivenMetadata.java — the funnel itself, the ONE legal direct call site +# (its javadoc mentions of the call live here too). +# 2. Any bare call whose line — or the line immediately above it — carries the marker +# "getMetadata-funnel-exempt": the write-path seams (INSERT / DELETE / MERGE sink, +# bind, row-level DML), temporarily exempt until the write-sharing step reroutes them +# through the funnel. Deleting a marker there auto-tightens the gate onto that site. +# 3. A no-argument getMetadata() call (RuntimeProfile's node probe, TestExternalCatalog's +# static map): a different method, not the Connector#getMetadata(ConnectorSession) SPI. +# 4. A comment line that merely names the call — never executable. +# +# The match is anchored to the call form (".getMetadata(" with an argument), NOT to +# getMetadataTableRows(...) or the API declaration. +# +# Self-test: tools/check-fecore-metadata-funnel.test.sh. +# +# Usage: +# tools/check-fecore-metadata-funnel.sh # default root fe/fe-core +# tools/check-fecore-metadata-funnel.sh # supplied root +# +# Exit code: +# 0 — no un-exempt bare calls +# 1 — at least one found (offending lines printed) +# 2 — search root not found + +set -e + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +DEFAULT_ROOT="${SCRIPT_DIR}/../fe/fe-core" +ROOT="${1:-${DEFAULT_ROOT}}" + +if [ ! -d "${ROOT}" ]; then + echo "check-fecore-metadata-funnel: search root not found: ${ROOT}" >&2 + exit 2 +fi + +# The funnel carrier — the ONE file allowed to call Connector#getMetadata directly. +FUNNEL_REL='datasource/plugin/PluginDrivenMetadata.java' + +# Inline marker for the write-path seams still bypassing the funnel (removed by the write-sharing step). +# It may sit on the call line or on the line directly above it (long call lines keep the marker above). +MARKER='getMetadata-funnel-exempt' + +# An invocation of getMetadata with an argument. Two forms so a call whose argument is wrapped onto the +# next line (open paren at end of line) is still caught, not just the single-line form: +ARG_SAMELINE='\.getMetadata\([[:space:]]*[^[:space:])]' # ".getMetadata(session" — arg on this line +ARG_WRAPPED='\.getMetadata\([[:space:]]*$' # ".getMetadata(" — arg on next line + +# Candidate lines: any ".getMetadata(" under main sources. Narrowed to real arg calls in the loop. +CANDIDATES=$(grep -rEn '\.getMetadata\(' "${ROOT}/src/main/java" 2>/dev/null || true) + +RESULT="" +if [ -n "${CANDIDATES}" ]; then + while IFS= read -r line; do + [ -z "${line}" ] && continue + # line = :: + file="${line%%:*}" + rest="${line#*:}" + lineno="${rest%%:*}" + code="${rest#*:}" + + # (4) comment line merely naming the call — never executable. + trimmed="${code#"${code%%[![:space:]]*}"}" + case "${trimmed}" in '*'*|'//'*|'/*'*) continue ;; esac + + # (3) no-argument getMetadata() — a different method, not the SPI call. + if ! { [[ "${code}" =~ ${ARG_SAMELINE} ]] || [[ "${code}" =~ ${ARG_WRAPPED} ]]; }; then + continue + fi + + # (1) the funnel file itself. + case "${file}" in *"${FUNNEL_REL}") continue ;; esac + + # (2) tracked write-path exemption marker, on the call line ... + case "${code}" in *"${MARKER}"*) continue ;; esac + # ... or on the line immediately above it. + if [ "${lineno}" -gt 1 ]; then + prev="$(sed -n "$((lineno - 1))p" "${file}" 2>/dev/null || true)" + case "${prev}" in *"${MARKER}"*) continue ;; esac + fi + + RESULT="${RESULT}${line}"$'\n' + done <<< "${CANDIDATES}" +fi +RESULT=$(printf '%s' "${RESULT}" | sed '/^$/d') + +if [ -n "${RESULT}" ]; then + echo "BARE Connector#getMetadata() calls in fe-core (must route through PluginDrivenMetadata.get):" >&2 + echo "${RESULT}" >&2 + echo "" >&2 + echo "A statement must reuse ONE ConnectorMetadata per catalog. Acquire it via" >&2 + echo "PluginDrivenMetadata.get(session, connector), not connector.getMetadata(session)." >&2 + echo "Write-path seams pending the write-sharing step carry a '${MARKER}' marker" >&2 + echo "(on the call line or the line directly above it)." >&2 + exit 1 +fi diff --git a/tools/check-fecore-metadata-funnel.test.sh b/tools/check-fecore-metadata-funnel.test.sh new file mode 100755 index 00000000000000..df68fd0decccdf --- /dev/null +++ b/tools/check-fecore-metadata-funnel.test.sh @@ -0,0 +1,160 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Self-test for tools/check-fecore-metadata-funnel.sh. +# +# The gate exits 0 on the real (already-routed) tree, so a controlled RED/GREEN fixture is the +# only way to prove it catches what it must and to lock the behavior against silent regression. +# Each seeded case targets one gate property: +# RED — a bare arg call in a normal file (no marker) (the core violation) +# RED — a bare call whose arg is WRAPPED to the next line (open paren at EOL must still catch) +# SILENT — the funnel file's own direct call + javadoc (whitelisted by path) +# SILENT — a marked call, marker on the call line (same-line exemption) +# SILENT — a marked call, marker on the line above (above-line exemption, for long lines) +# SILENT — a no-argument getMetadata() (different method, not the SPI call) +# SILENT — getMetadataTableRows(...) (method-name boundary) +# SILENT — a comment line naming the call (never executable) +# Plus: exit 0 on a clean tree, and the marker is load-bearing (strip it => the call is flagged). +# +# Usage: bash tools/check-fecore-metadata-funnel.test.sh # exit 0 = pass, 1 = fail + +set -u + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +GATE="${SCRIPT_DIR}/check-fecore-metadata-funnel.sh" + +FX="$(mktemp -d)" +trap 'rm -rf "${FX}"' EXIT + +SRC="${FX}/src/main/java/org/apache/doris" +mkdir -p "${SRC}/datasource/plugin" "${SRC}/read" "${SRC}/write" "${SRC}/misc" + +# SILENT: the funnel — the ONE legal direct call site, plus a javadoc mention of the call. +cat > "${SRC}/datasource/plugin/PluginDrivenMetadata.java" <<'EOF' +package org.apache.doris.datasource.plugin; +/** + * Memoizes {@code connector.getMetadata(session)} once per statement. + */ +public final class PluginDrivenMetadata { + static Object get(Object session, Object connector) { + return scope.getOrCreateMetadata(key, () -> connector.getMetadata(session)); + } +} +EOF + +# RED: bare arg call, no marker, not the funnel. +cat > "${SRC}/read/Reader.java" <<'EOF' +package org.apache.doris.read; +public class Reader { + void f() { + Object m = connector.getMetadata(session); + } +} +EOF + +# RED: bare call whose argument is wrapped onto the next line (open paren at end of line). +cat > "${SRC}/read/Wrapped.java" <<'EOF' +package org.apache.doris.read; +public class Wrapped { + void f() { + Object m = connector.getMetadata( + session); + } +} +EOF + +# SILENT: write-path call, marker on the SAME line. +cat > "${SRC}/write/WriterTrailing.java" <<'EOF' +package org.apache.doris.write; +public class WriterTrailing { + void f() { + Object m = connector.getMetadata(session); // getMetadata-funnel-exempt: write path + } +} +EOF + +# SILENT: write-path call, marker on the line ABOVE (how long call lines carry it). +cat > "${SRC}/write/WriterAbove.java" <<'EOF' +package org.apache.doris.write; +public class WriterAbove { + void f() { + // getMetadata-funnel-exempt: write path, rerouted through the funnel in the write-sharing step + Object m = catalog.getConnector().getMetadata(session); + } +} +EOF + +# SILENT: no-arg getMetadata(), getMetadataTableRows(...), and a comment naming the call. +cat > "${SRC}/misc/Misc.java" <<'EOF' +package org.apache.doris.misc; +public class Misc { + void f() { + int id = (int) node.getMetadata(); + Object rows = table.getMetadataTableRows(session); + // comment: connector.getMetadata(session) must stay silent + Object keys = catalogProvider.getMetadata().keySet(); + } +} +EOF + +FAILED=0 +fail() { echo "FAIL: $1"; FAILED=1; } + +# ---- run 1: mixed fixture -> exactly the two RED cases flagged ---- +OUT="$(bash "${GATE}" "${FX}" 2>&1)"; EC=$? +REPORTED="$(printf '%s\n' "${OUT}" | grep -E "^${FX}.*:[0-9]+:" || true)" +N="$(printf '%s\n' "${REPORTED}" | grep -c 'getMetadata' || true)" + +[ "${EC}" -eq 1 ] || fail "expected exit 1 (violations present), got ${EC}" +[ "${N}" -eq 2 ] || fail "expected exactly 2 reported violations, got ${N}"$'\n'"${REPORTED}" + +must_report() { printf '%s\n' "${REPORTED}" | grep -qF "$1" || fail "violation NOT reported: $1"; } +must_report 'read/Reader.java:' # core: bare arg call, no marker +must_report 'read/Wrapped.java:' # wrapped-arg call (open paren at EOL) + +must_not_report() { + printf '%s\n' "${REPORTED}" | grep -qF "$1" && fail "should NOT be reported: $1" || true +} +must_not_report 'PluginDrivenMetadata.java:' # funnel (whitelisted by path) +must_not_report 'WriterTrailing.java:' # marker on call line +must_not_report 'WriterAbove.java:' # marker on line above +must_not_report 'Misc.java:' # no-arg / getMetadataTableRows / comment + +# ---- run 2: remove the RED cases -> clean tree exits 0 (proves all SILENT cases stay silent) ---- +rm -f "${SRC}/read/Reader.java" "${SRC}/read/Wrapped.java" +bash "${GATE}" "${FX}" >/dev/null 2>&1 && CLEAN_EC=0 || CLEAN_EC=$? +[ "${CLEAN_EC}" -eq 0 ] || fail "expected exit 0 on clean tree (only funnel/marked/no-arg/comment), got ${CLEAN_EC}" + +# ---- run 3: the marker is load-bearing -> strip it and both write calls are flagged ---- +sed -i 's/getMetadata-funnel-exempt/marker-removed-here/' \ + "${SRC}/write/WriterTrailing.java" "${SRC}/write/WriterAbove.java" +OUT3="$(bash "${GATE}" "${FX}" 2>&1)"; EC3=$? +REP3="$(printf '%s\n' "${OUT3}" | grep -E "^${FX}.*:[0-9]+:" || true)" +N3="$(printf '%s\n' "${REP3}" | grep -c 'getMetadata' || true)" +[ "${EC3}" -eq 1 ] || fail "expected exit 1 after stripping markers, got ${EC3}" +[ "${N3}" -eq 2 ] || fail "expected 2 flagged after stripping markers, got ${N3}"$'\n'"${REP3}" +printf '%s\n' "${REP3}" | grep -qF 'WriterTrailing.java:' || fail "same-line call not flagged after marker strip" +printf '%s\n' "${REP3}" | grep -qF 'WriterAbove.java:' || fail "above-line call not flagged after marker strip" + +if [ "${FAILED}" -eq 0 ]; then + echo "PASS: funnel gate catches bare (incl. wrapped) calls; funnel/marker/no-arg/comment stay silent; marker is load-bearing." + exit 0 +fi +echo "---- run1 output ----"; printf '%s\n' "${OUT}" +exit 1 From 98d3dae6d4792ec4b6427ffbb7086e755be957a7 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 20:52:44 +0800 Subject: [PATCH 35/47] [doc](catalog) per-statement metadata: read-keystone closeout (anti-drift gate) + STEP 2 handoff RD-1 (read keystone) marked complete: C1 foundation + C2 close wiring + C3 reroute + scan-node field + background read-through + the anti-drift gate. Progress log records the gate landing (grep universe, rule set, verification); handoff re-points the next session at the HMS heterogeneous-gateway sibling fan-out (STEP 2 / RD-2), including the note that the sibling funnel lives in fe-connector-hive and must go through session.getStatementScope().getOrCreateMetadata (not fe-core's PluginDrivenMetadata) to stay off the connector-imports gate. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../per-statement-table-owner-port/HANDOFF.md | 59 ++++++++++--------- .../progress.md | 13 ++++ .../tasklist.md | 2 +- 3 files changed, 44 insertions(+), 30 deletions(-) diff --git a/plan-doc/per-statement-table-owner-port/HANDOFF.md b/plan-doc/per-statement-table-owner-port/HANDOFF.md index dd85d3e2e41a60..9f6042f1f63c88 100644 --- a/plan-doc/per-statement-table-owner-port/HANDOFF.md +++ b/plan-doc/per-statement-table-owner-port/HANDOFF.md @@ -5,40 +5,40 @@ --- -# 🆕 本轮（2026-07-19 session 4）已完成：**落地读取侧改道 + 后台读穿纠正（读取键石主体收官）** +# 🆕 本轮（2026-07-19 session 5）已完成：**防漂移门禁落地 → 读取键石（STEP 1 / RD-1）完全收官** ## 一句话结果 -- 动手前先过对抗式核对关（workflow `wf_6e2967a9-1a2`），把改道表逐条对当前代码行号重核 → `designs/C3-read-reroute-verified-checklist.md`：fe-core 连接器 `getMetadata` 工厂缝**共 66 处**完整切四类（相加=66、零遗漏/重叠/未分类）。两处偏差经用户拍板。 -- **已实现（全绿，未提交）**： - - **助手** `PluginDrivenExternalCatalog.buildCrossStatementSession()`：镜像 `buildConnectorSession`（保留凭证），只多强制 `withStatementScope(NONE)`。 - - **9 处后台读穿**（含名字映射两缝 `fromRemoteDatabaseName`/`fromRemoteTableName` 经拍板归入）：改用助手 + 走统一入口 `PluginDrivenMetadata.get`；**含 `fetchRowCount` ANALYZE 隐患修复**（不必单改统计模块）。源码证实「NONE 走漏斗」与裸直调**字节等价**。 - - **49 处改道**：读/DDL/命令/多版本从 `connector.getMetadata(session)` 改调 `PluginDrivenMetadata.get(session, connector)`。 - - **扫描节点存字段**：`volatile cachedMetadata` + `metadata()`；8 处 per-method 改调它，静态 `create` 直调入口。 - - **测试适配**（workflow `wf_fbb60841-365`，11 agents 并行）：补 `getStatementScope()→NONE` stub + 测试替身补 `buildCrossStatementSession` override；新守门 `ConnectorSessionImplTest.explicitNoneStatementScopeWinsOverLiveContext`。 -- **验证**：目标测试 247 全绿（先红 130 error+30 fail）+ ConnectorSessionImplTest 18 绿 + fe-core checkstyle 0 违规 + 主编译 SUCCESS。 +- 用户拍板方案 A（现在就上门禁锁死读取侧、写入 8 处显式豁免）。 +- **已实现（全绿，已提交 `b2d147998d1`）**： + - 新增 `tools/check-fecore-metadata-funnel.sh`（bash grep 门禁，仿 `check-connector-imports.sh`）+ 自测 `.test.sh`：扫 `fe/fe-core/src/main/java`，禁裸 `Connector#getMetadata(session)`；放行=①funnel 文件 `PluginDrivenMetadata.java`（含 javadoc）②带 `getMetadata-funnel-exempt` 标记的行（call 行**或其上一行**）③无参 `getMetadata()`（异方法）④注释行。正则双形（同行参数 + 换行参数），不误伤 `getMetadataTableRows`/API 定义。 + - 8 处写入裸调各加一行上置标记注释（103 字）；挂入 `fe/fe-core/pom.xml` validate 阶段 exec（与 fe-connector 门禁同深度同范式）。 +- **验证**：自测 PASS（10 项含核心/换行/白名单/同行+上行标记/无参/边界/注释/退出码/标记承重）；门禁真实树 exit 0、8 处未标记 exit 1（都证过）；fe-core checkstyle 0 违规；`mvn -pl fe-core validate` 实跑触发 exec + BUILD SUCCESS。 +- **读取键石收官**：C1 地基 + C2 关闭 + C3 改道 + 扫描存字段 + 后台读穿 + 防漂移门禁全落地（RD-1 = ✅）。 --- -# ➡️ 下一个 session = **读取键石收官（防漂移门禁）→ 然后 HMS 异构网关兄弟扇出** +# ➡️ 下一个 session = **HMS 异构网关兄弟扇出（STEP 2 / RD-2）** ## 第一件事（先读） -1. 读 `designs/C3-read-reroute-verified-checklist.md`（已实施；§1 四类分区、§2 强制 NONE 机制、用户两处拍板）。 -2. 读 `designs/P1-implementation-design.md` §5（HMS sibling 扇出蓝图：三连接器共享 catalogId → key 加属主 label）+ §8（arch 门禁蓝图）。 -3. 读架构记忆 `iceberg-table-resolution-cache-scoping`、`catalog-spi-hms-hiveversionutil-gate-false-positive`（门禁误报先例）。 - -## A. 收官：防漂移门禁（读取键石最后一步，小） -- 目标：加一道构建期门禁（bash grep，仿 `tools/check-connector-imports.sh` 风格 + 自测），确保 fe-core 里裸 `connector.getMetadata(` **只允许出现在** `PluginDrivenMetadata.java`（漏斗自身的 factory lambda）。 -- **本轮已把读穿 loader 也走统一入口 → 门禁零例外**（无需白名单）。当前 fe-core 里合法裸调用只剩：`PluginDrivenMetadata.java:54`（漏斗 lambda）+ 写路径 8 处（`PhysicalPlanTranslator`×2 / `BindSink`×2 / `IcebergRowLevelDmlTransform` / `PhysicalIcebergMergeSink` / `PluginDrivenInsertExecutor` / `PluginDrivenExternalTable.resolveWriteTargetHandle:133`）。 -- **决策点（交用户）**：写路径 8 处现仍裸直调（留写入共用步骤）。门禁要么(a)现在只锁"非写文件"、写文件暂豁免（写入共用步骤再收）；要么(b)推迟整个门禁到写入共用步骤后一次落。建议(a)：现在就锁读侧防漂移，写文件显式列为"暂待写入步骤"豁免。落点见 P1-design §8（checkstyle Regexp 或 fe-connector pom 的独立 exec）。 - -## B. 下一大步：HMS 异构网关兄弟扇出（STEP 2 / P1-design §5） -- 网关已 LIVE（hms 在 `SPI_READY_TYPES`）。三连接器（hive/iceberg/hudi）**共享一个 catalogId** → 若统一入口 key 只用 catalogId 会把三者塌成一个 metadata、派错。 -- **修**：①入口 key 加属主 label（`"metadata:"+catalogId+":"+owner`，owner∈{hive,iceberg,hudi}，在 `resolveSiblingOwner` 命中臂后取 label）；②**兄弟 getMetadata 也进入口**（`siblingMetadata`/`icebergSiblingMetadata`/`hudiSiblingMetadata`）；③只存/返回 `ConnectorMetadata` 接口不 cast；④补异构网关 e2e（`external_table_p2/refactor_catalog_param`，对齐 `hms-iceberg-delegation-needs-e2e`）。 -- 注意：sibling 扇出在**连接器内部**（fe-connector-hive），fe-core arch 门禁看不见 → 由连接器侧单测守（仿 `HiveConnectorSiblingTest`）。 +1. 读 `designs/expanded-scope-phasing-and-security-decisions.md` §1（读写共用忠实 Trino）+ §2（后台读穿），尤其 **§1 末：兄弟入口须覆盖 `beginTransaction` 路由，不仅 getMetadata**。 +2. 读 `designs/P1-implementation-design.md` §5（HMS sibling 扇出蓝图：三连接器共享 catalogId → key 加属主 label；~43 处 per-handle 转发；`resolveSiblingOwner` 三路派发；DCL sibling 身份稳定）。 +3. 读架构记忆 `catalog-spi-plugin-tccl-classloader-gotcha`（ThriftHmsClient/HiveConf split-brain 四 locus）、`hms-iceberg-delegation-needs-e2e`（每项 iceberg-on-HMS 新能力都要配 e2e）、`iceberg-table-resolution-cache-scoping`。 + +## A. 动码前先 grounding + 出中文方案待确认（本任务铁律：设计先行） +- 网关已 LIVE（hms 在 `SPI_READY_TYPES`，commit `83585fd5097`）。sibling 扇出全在**连接器内部**（fe-connector-hive）→ fe-core arch 门禁看不见，由连接器侧单测守（仿 `HiveConnectorSiblingTest`）。 +- 对当前代码逐点核实（会漂移）：`HiveConnectorMetadata.getTableHandle` 按格式转发 + `siblingMetadata`/`icebergSiblingMetadata`/`hudiSiblingMetadata`（各转发点当前行号）；`HiveConnector.resolveSiblingOwner`（三路 `ownsHandle`）；`beginTransaction` 路由点（`HiveConnectorMetadata:1854` 附近，核实）；`createSiblingConnector` 传 `this` context → 三连接器同 catalogId 的证据（`DefaultConnectorContext`）。 +- 出中文方案（不引任务代号）：为何 catalogId 单独做 key 会把三连接器塌成一个 metadata、派错；修法四点（见 B）；e2e 落点。**待用户确认后再改代码。** + +## B. 实现要点（分期定稿 §1 末 + P1-design §5） +1. **funnel key 加属主 label**：`"metadata:"+catalogId+":"+owner`，`owner∈{hive,iceberg,hudi}`，**在 `resolveSiblingOwner` 命中臂后取 label**（非预解析、非 identityHashCode）。 +2. **兄弟 getMetadata 及 beginTransaction 都进同一入口**：`siblingMetadata`/`icebergSiblingMetadata`/`hudiSiblingMetadata` + 写事务 mint 点，先解析属主 Connector（照旧可 fail-loud），再 `scope.getOrCreateMetadata(key(catalogId,ownerLabel), () -> owner.getMetadata(session))`；getTableHandle 的 by-TYPE 转发与后续 per-handle 转发共享一个 sibling metadata。 +3. **只存/返回 `ConnectorMetadata` 接口，绝不 cast 具体类型**（跨 loader CCE）。 +4. **补异构网关 e2e**（`external_table_p2/refactor_catalog_param`，对齐 `hms-iceberg-delegation-needs-e2e`）：异构 HMS 目录跑 INSERT/DELETE/MERGE/ALTER/EXECUTE 断言与独立 iceberg 目录同表同结果。 +- 连接器侧的 funnel 入口 = fe-core 的 `PluginDrivenMetadata.get`？注意：sibling 扇出在 fe-connector-hive 内，**不能 import fe-core**（`check-connector-imports` 门禁会挂）。→ 入口须是**连接器可见的 SPI**（`ConnectorStatementScope.getOrCreateMetadata` 已在 fe-connector-api，sibling 直接用 `session.getStatementScope().getOrCreateMetadata(key, factory)`，不经 fe-core 的 `PluginDrivenMetadata`）。**grounding 时确认这条路径**（label key 构造放连接器侧）。 ## 后续步（详见分期定稿 §4） -- **STEP 3 写入共用**：3a 无状态写点（含 `resolveWriteTargetHandle`）改道进入口；3b `ConnectorTransaction` 归属上移 `CatalogStatementTransaction`。闸门=读写身份等价 + 保留 hive 起写刷新 + 保留 tx↔session 绑定。纠缠点：iceberg 起写复用 `IcebergStatementScope.sharedTable`（P2 计划删）→ 3b 时定先后。 -- **STEP 4 缓存隔离（安全）**：`getIdentityShardKey()` SPI → iceberg 三投影缓存分片 + fe-core 表结构缓存 bypass(先)/分片(后) + 防漂移门禁；随 STEP 2 属主键覆盖异构网关；越权 e2e + 威胁模型签字。 +- **STEP 3 写入共用（RD-3）**：3a 无状态写点（含 `resolveWriteTargetHandle`）改道进入口（删对应 `getMetadata-funnel-exempt` 标记 → 门禁自动收紧）；3b `ConnectorTransaction` 归属上移 `CatalogStatementTransaction`。闸门=读写身份等价 + 保留 hive 起写刷新（`HiveConnectorTransaction.beginWrite` 当场 getTable）+ 保留 tx↔session 绑定。纠缠点：iceberg 起写复用 `IcebergStatementScope.sharedTable`（P2 计划删）→ 3b 时定先后。 +- **STEP 4 缓存隔离（安全，RD-4）**：`getIdentityShardKey()` SPI → iceberg 三投影缓存分片 + fe-core 表结构缓存 bypass(先)/分片(后) + 防漂移门禁；随 STEP 2 属主键覆盖异构网关；越权 e2e + 威胁模型签字。 ## 关闭接线残留风险（carry-forward，详见分期定稿 §6） 1. 取消/超时非硬栅栏（`SplitAssignment.stop` 只置标志）——既有共担，P1 no-op 关闭下无实害，**关闭做实事前须硬化**。 @@ -47,16 +47,17 @@ 4. **🔴 TCCL 自钉扎（硬前置）**：连接器 `close()` 做实事（P2+）后，主关闭回调 + 兜底两处都须把 TCCL 钉到插件 classloader（连接器 `close()` 自钉扎首选）。 ## 铁律 / 闸门提醒 -- 用户 2026-07-19 已豁免铁律 A（fe-core 只减不增）。仍守：连接器 connector-agnostic（作用域值 Object）；作用域跨用户即泄漏（STEP 4 威胁模型）。 +- 用户 2026-07-19 已豁免铁律 A（fe-core 只减不增）。仍守：连接器 connector-agnostic（作用域值 Object）；作用域跨用户即泄漏（STEP 4 威胁模型）；**fe-connector-hive 不得 import fe-core**（门禁）。 - **动码前探测并发活动**（git log/status + maven 进程 + 近 90s mtime），发现活跃即停手（`concurrent-sessions-shared-worktree-hazard`）。 +- 本任务设计先行：调研设计阶段结束、正式改代码前，先把方案用中文详述待用户确认。 --- # 🗂 遗留 / 关联 -- 分期定稿（现行主线）：`designs/expanded-scope-phasing-and-security-decisions.md`（含 §6 进展 + 残留风险）。 +- 分期定稿（现行主线）：`designs/expanded-scope-phasing-and-security-decisions.md`（含 §1 读写共用、§2 后台、§4 分期、§6 残留风险）。 - STEP 1 蓝图：`designs/P1-implementation-design.md`（§2 改道表；§4 两层关闭；§5 HMS sibling；§8 arch 门禁）。 - C3 核对清单（已实施）：`designs/C3-read-reroute-verified-checklist.md`。 - 目标架构全景：`designs/trino-parity-metadata-redesign-design.md`。 -- 本轮 workflow：`wf_6e2967a9-1a2`（C3 逐缝核对+对抗）、`wf_fbb60841-365`（测试适配 11 文件）。前轮 `wf_8b907b93-e9f`/`wf_9250330b-e81`。 -- 架构记忆：`iceberg-table-resolution-cache-scoping`。 +- 门禁：`tools/check-fecore-metadata-funnel.sh`（+ `.test.sh`），fe-core pom validate exec。 +- 架构记忆：`catalog-spi-plugin-tccl-classloader-gotcha`、`hms-iceberg-delegation-needs-e2e`、`iceberg-table-resolution-cache-scoping`。 - e2e 一律留连接器进 `SPI_READY_TYPES` 切换阶段统一补；异构网关 e2e 随 STEP 2/4。 diff --git a/plan-doc/per-statement-table-owner-port/progress.md b/plan-doc/per-statement-table-owner-port/progress.md index 24de2745e12ac9..76783bbafe331d 100644 --- a/plan-doc/per-statement-table-owner-port/progress.md +++ b/plan-doc/per-statement-table-owner-port/progress.md @@ -72,3 +72,16 @@ - **验证**：目标测试 247 全绿（先红 130 error+30 fail → 修后 0/0）+ ConnectorSessionImplTest 18 绿 + fe-core checkstyle BUILD SUCCESS 零违规 + fe-core 主编译 BUILD SUCCESS。 - **提交口径**：4 个逻辑子步（助手/读穿/改道/扫描存字段）在 2 个共享文件里交织，且每提交须保持绿（测试适配须随产品改动同提），逐 hunk 拆分需交互式 git（本环境不支持）→ 按 C1/C2 先例=1 个绿代码提交（含产品+测试适配+守门）+ 1 个文档提交，代码提交体内枚举 4 子步。 - **下一步**：见 HANDOFF —— 读取键石剩防漂移门禁（形态=统一入口零例外）；下一大步 = HMS 异构网关兄弟扇出（键含属主 label + 兄弟 getMetadata/起事务进同一入口 + 异构网关 e2e）。 + +## 2026-07-19 — session 5：防漂移门禁落地（读取键石完全收官） + +- **用户拍板（动码前，方案 A）**：现在就上门禁锁死读取侧、写入 8 处显式豁免（写入共用步骤再收）；否决"推迟整个门禁到写入步骤后一次落"。 +- **动手前取证**：对当前树 grep `.getMetadata(` 全域 = 21 行 = 9 处无参（`TestExternalCatalog`×8 静态 Map + `RuntimeProfile.node.getMetadata()`，均 `src/main`）+ 3 处 funnel javadoc + 1 处 funnel 真调 + **8 处写入裸调**。读取侧已"零裸调"（49 处 C3 已改道走 `PluginDrivenMetadata.get`）。 +- **落地（commit `b2d147998d1`）**： + - 新增 `tools/check-fecore-metadata-funnel.sh`（bash grep 门禁，仿 `check-connector-imports.sh`）+ 自测 `.test.sh`。规则：扫 `fe/fe-core/src/main/java`，禁裸 `Connector#getMetadata(session)`；**放行**=①funnel 文件 `datasource/plugin/PluginDrivenMetadata.java`（含其 javadoc）②带 `getMetadata-funnel-exempt` 标记的行（**call 行或其上一行**，兼容长行）③无参 `getMetadata()`（异方法）④注释行。正则双形匹配（同行参数 + 换行到下一行的参数），锚定调用形不误伤 `getMetadataTableRows`/API 定义。 + - 8 处写入裸调各加一行上置标记注释（103 字，全 <120）：PhysicalPlanTranslator×2 / BindSink×2 / PluginDrivenInsertExecutor / IcebergRowLevelDmlTransform / PhysicalIcebergMergeSink / PluginDrivenExternalTable(resolveWriteTargetHandle)。删标记即自动收紧到该处。 + - 挂入 `fe/fe-core/pom.xml` validate 阶段 exec（`${project.basedir}/../../tools/...`，与 fe-connector 门禁同深度同范式）。 +- **验证**：自测 PASS（含核心裸调、换行参数、funnel 白名单、同行/上行标记、无参跳过、`getMetadataTableRows` 边界、注释跳过、退出码、标记可承重 10 项）；门禁对真实树 exit 0，对 8 处未标记 exit 1（都证过）；fe-core checkstyle 0 违规；`mvn -pl fe-core validate` 实跑触发门禁 exec + BUILD SUCCESS。 +- **读取键石（RD-1 / STEP 1）至此完全收官**：C1 地基 + C2 关闭 + C3 改道 + 扫描存字段 + 后台读穿 + 防漂移门禁全落地。 +- **未提交的第三方无关文件不动**（stray untracked `fe/.mvn/maven.config` 等非本轮产物，只 stage 本轮 9 文件）。 +- **下一步**：见 HANDOFF —— 下一大步 = HMS 异构网关兄弟扇出（RD-2 / STEP 2），动码前先按分期定稿 §1/§2 + P1-design §5 对当前代码做 grounding recon 并把方案用中文详述待用户确认。 diff --git a/plan-doc/per-statement-table-owner-port/tasklist.md b/plan-doc/per-statement-table-owner-port/tasklist.md index b19652d4032a9c..3c4e503b3ddcce 100644 --- a/plan-doc/per-statement-table-owner-port/tasklist.md +++ b/plan-doc/per-statement-table-owner-port/tasklist.md @@ -10,7 +10,7 @@ | ID | 步 | 状态 | 内容 | 依赖 | commit | |---|---|---|---|---|---| -| RD-1 | STEP 1 读取键石 | 🚧 进行中（C1+C2+C3 已落地，仅剩防漂移门禁） | C1 地基 ✅ + C2 关闭接线 ✅ + C3 读/扫描/DDL/MVCC 改道（49 处）✅ + 扫描节点存字段 ✅ + 后台 loader 显式 NONE 读穿（9 处，含名字映射两缝）✅ + 修 fetchRowCount ANALYZE 隐患 ✅ + `buildCrossStatementSession` 助手 ✅ + 守门测试 ✅（247+18 绿、checkstyle 0）。剩：防漂移门禁（bash grep，形态=统一入口零例外）⏳ | — | C1 `5b7312f9d1f` · C2 `12f3e95239b` · C3 待提交 | +| RD-1 | STEP 1 读取键石 | ✅ 完成 | C1 地基 ✅ + C2 关闭接线 ✅ + C3 读/扫描/DDL/MVCC 改道（49 处）✅ + 扫描节点存字段 ✅ + 后台 loader 显式 NONE 读穿（9 处，含名字映射两缝）✅ + 修 fetchRowCount ANALYZE 隐患 ✅ + `buildCrossStatementSession` 助手 ✅ + 守门测试 ✅（247+18 绿、checkstyle 0）+ **防漂移门禁**（bash grep，读取侧零例外、写入 8 处 `getMetadata-funnel-exempt` 标记豁免；自测+validate exec）✅ | — | C1 `5b7312f9d1f` · C2 `12f3e95239b` · C3 `eecb390c4ac` · 门禁 `b2d147998d1` | | RD-2 | STEP 2 HMS 兄弟扇出 | ⏳ 待启动 | 键=(catalogId, ownerLabel)；兄弟 getMetadata **及 beginTransaction** 进同一 funnel；异构网关 e2e | RD-1 | | | RD-3 | STEP 3 写入共用 | ⏳ 待启动 | 3a 无状态写点改道进 funnel；3b `ConnectorTransaction` 归属上移 `CatalogStatementTransaction` + commit/rollback vs closeAll 顺序。闸门=读写身份等价 + 保留 hive 起写刷新 + 保留 tx↔session 绑定 | RD-1,RD-2 | | | RD-4 | STEP 4 缓存隔离（安全 track） | ⏳ 待启动（**list≠load 已确认=真实越权**） | `getIdentityShardKey()` SPI → iceberg 三投影缓存 Key 分片 + fe-core 表结构缓存 bypass(先)/分片(后) + 防漂移门禁；随 STEP 2 属主键覆盖异构网关；越权 e2e | 威胁模型签字 + RD-2 | | From 5e6fcb131d3e1b5b65bc774c14a8714b0cc56df0 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 22:20:33 +0800 Subject: [PATCH 36/47] [feat](catalog) fe-connector-hive: memoize HMS-gateway sibling metadata per statement The Hive heterogeneous gateway forwards iceberg/hudi-on-HMS operations to embedded sibling connectors. It obtained a fresh sibling ConnectorMetadata on every forward, so one statement rebuilt a table's sibling metadata a dozen-plus times. Route the four sibling-metadata acquisition sites through the existing per-statement scope so a statement reuses ONE sibling metadata per owner (mirroring fe-core's own funnel). - Add SiblingOwner{connector,label}. HiveConnector.resolveSiblingOwnerLabeled yields the owner label from its matched ownsHandle arm (no force-build, no identity comparison). resolveSiblingOwner now delegates to it, so the three connector-level provider seams are unchanged. - HiveConnectorMetadata.memoizedSiblingMetadata keys the scope as "metadata:"+catalogId+":"+label. The three connectors share one catalogId, so the label keeps their metadata entries distinct (a plain catalogId key would collapse them and misroute). The three helpers (icebergSiblingMetadata / hudiSiblingMetadata by TYPE, siblingMetadata by HANDLE) and the getTableSchema capability-reflection stray route through it; beginTransaction(session,handle) is covered for free because it forwards via siblingMetadata. By-TYPE and by-HANDLE mint the same label for one owner, so the getTableHandle divert and later per-handle forwards reuse one instance. - Under NONE scope the factory runs on every call: byte-identical to before. Only fe-connector-api types are used (no fe-core import); the foreign sibling is never cast. Tests: five funnel assertions (one build across many forwards incl. the write txn; NONE rebuilds each call; by-TYPE == by-HANDLE key; cross-catalog isolation; iceberg / hudi isolated by label). Existing sibling suites now pass a NONE-scope session (the forward path dereferences the scope); no forwarding / routing / fail-loud assertion is weakened. Copies TestStatementScope + a ScopeSession test double into the hive test tree (connector tests cannot import fe-core / iceberg). The heterogeneous-gateway e2e (INSERT/DELETE/MERGE/ALTER/EXECUTE on an iceberg-on-HMS table vs a standalone iceberg catalog) is deferred to the later unified delegation-e2e pass. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../doris/connector/hive/HiveConnector.java | 29 +- .../connector/hive/HiveConnectorMetadata.java | 68 +++-- .../doris/connector/hive/SiblingOwner.java | 60 ++++ ...onnectorMetadataSiblingDelegationTest.java | 268 ++++++++++++------ ...onnectorMetadataTableHandleDivertTest.java | 26 +- .../HiveConnectorThreeWayRoutingTest.java | 9 +- .../doris/connector/hive/ScopeSession.java | 89 ++++++ .../connector/hive/TestStatementScope.java | 40 +++ 8 files changed, 465 insertions(+), 124 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/SiblingOwner.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/ScopeSession.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/TestStatementScope.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 7846b13bd19ef0..0a2d4ea32a21c1 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -169,16 +169,18 @@ private static String rootCauseMessage(Throwable t) { */ HiveConnectorMetadata newMetadata(HmsClient client) { return new HiveConnectorMetadata(client, properties, context, - this::getOrCreateIcebergSibling, this::getOrCreateHudiSibling, this::resolveSiblingOwner, + this::getOrCreateIcebergSibling, this::getOrCreateHudiSibling, this::resolveSiblingOwnerLabeled, fileListingCache); } /** - * Resolves which embedded sibling connector OWNS a foreign (non-hive) table handle, for the per-handle - * gateway seams (the connector-level {@code get*Provider(handle)} below and the ~34 guard-and-forward - * methods in {@link HiveConnectorMetadata}). Asks each sibling's {@link Connector#ownsHandle} — the sibling - * tests its OWN in-loader handle type, which is invisible to the hive loader across the plugin split, so the - * gateway can never {@code instanceof} the foreign handle itself. + * Resolves which embedded sibling connector OWNS a foreign (non-hive) table handle AND its stable owner label, + * for the per-handle gateway seams (the connector-level {@code get*Provider(handle)} below and the + * guard-and-forward methods in {@link HiveConnectorMetadata}). Asks each sibling's {@link Connector#ownsHandle} + * — the sibling tests its OWN in-loader handle type, which is invisible to the hive loader across the plugin + * split, so the gateway can never {@code instanceof} the foreign handle itself. The MATCHED ARM supplies the + * label ({@link SiblingOwner#ICEBERG_LABEL} / {@link SiblingOwner#HUDI_LABEL}), so a per-handle forward keys the + * per-statement metadata funnel by owner without a force-build or an identity comparison against the suppliers. * *

    Consults only ALREADY-BUILT siblings (a plain field read, never {@code getOrCreate*}). The owning * sibling is always already built: a foreign handle can only originate from {@code getTableHandle}'s divert, @@ -187,19 +189,28 @@ HiveConnectorMetadata newMetadata(HmsClient client) { * plugin must still route its hudi handles without building an iceberg sibling. Fails loud when no built * sibling owns the handle (an orphan handle is a bug, not a route), naming the catalog. */ - private Connector resolveSiblingOwner(ConnectorTableHandle handle) { + SiblingOwner resolveSiblingOwnerLabeled(ConnectorTableHandle handle) { Connector iceberg = icebergSibling; if (iceberg != null && iceberg.ownsHandle(handle)) { - return iceberg; + return new SiblingOwner(iceberg, SiblingOwner.ICEBERG_LABEL); } Connector hudi = hudiSibling; if (hudi != null && hudi.ownsHandle(handle)) { - return hudi; + return new SiblingOwner(hudi, SiblingOwner.HUDI_LABEL); } throw new DorisConnectorException("Cannot route a foreign table handle in catalog '" + context.getCatalogName() + "': no embedded sibling connector owns it"); } + /** + * The owning sibling {@link Connector} alone (label dropped) for the connector-level per-handle provider seams + * ({@code get*Provider(handle)} below), which route by owner but never touch the metadata funnel. Delegates to + * {@link #resolveSiblingOwnerLabeled} so the 3-way ownsHandle dispatch has a single implementation. + */ + private Connector resolveSiblingOwner(ConnectorTableHandle handle) { + return resolveSiblingOwnerLabeled(handle).connector(); + } + @Override public ConnectorScanPlanProvider getScanPlanProvider() { return new HiveScanPlanProvider(getOrCreateClient(), properties, context, readTxnManager, fileListingCache); diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 9e8ea66551561b..07f50ca76ca4a2 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -192,7 +192,7 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // The by-handle owner resolver installed by the 3-arg constructor (hive-only construction). Invoked only when // a NON-hive handle reaches a per-handle guard-and-forward site — which such a construction never does — so it // fails loud instead of NPEing. - private static final Function NO_SIBLING_OWNER = handle -> { + private static final Function NO_SIBLING_OWNER = handle -> { throw new DorisConnectorException("no sibling connector configured for this hive metadata"); }; @@ -218,7 +218,7 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // guard-and-forward methods below. Backed by HiveConnector.resolveSiblingOwner (a 3-way ownsHandle dispatch // over the ALREADY-BUILT iceberg / hudi siblings). Used ONLY through the parent-first Connector / // ConnectorMetadata interfaces — the owning sibling's concrete types are never cast here (cross-loader CCE). - private final Function siblingOwnerResolver; + private final Function siblingOwnerResolver; // Connector-owned directory-listing cache, shared with the scan provider so estimateDataSizeByListingFiles // (the periodic ExternalRowCountCache refresh source) reuses listings a scan warmed and vice versa. Injected // by HiveConnector.newMetadata as the SAME instance; the convenience constructors below build a private @@ -232,7 +232,7 @@ public HiveConnectorMetadata(HmsClient hmsClient, Map properties public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context, Supplier icebergSiblingSupplier, Supplier hudiSiblingSupplier, - Function siblingOwnerResolver) { + Function siblingOwnerResolver) { this(hmsClient, properties, context, icebergSiblingSupplier, hudiSiblingSupplier, siblingOwnerResolver, new HiveFileListingCache(properties)); } @@ -240,7 +240,7 @@ public HiveConnectorMetadata(HmsClient hmsClient, Map properties public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context, Supplier icebergSiblingSupplier, Supplier hudiSiblingSupplier, - Function siblingOwnerResolver, + Function siblingOwnerResolver, HiveFileListingCache fileListingCache) { this.hmsClient = hmsClient; this.properties = properties; @@ -251,46 +251,69 @@ public HiveConnectorMetadata(HmsClient hmsClient, Map properties this.fileListingCache = fileListingCache; } + /** + * Obtains the sibling's {@link ConnectorMetadata} through the per-statement funnel: within one statement, the + * first forward for a given owner runs {@code owner.getMetadata(session)} and every later forward (read / scan + * / DDL / MVCC / the per-handle {@code beginTransaction} open) reuses that ONE memoized instance — mirroring + * fe-core's own {@code PluginDrivenMetadata} funnel for a plain connector (the sibling's heavy catalog/caches + * live on the single memoized sibling CONNECTOR regardless of this). The key is + * {@code "metadata:" + catalogId + ":" + ownerLabel}: the three connectors of a heterogeneous gateway + * (hive + iceberg + hudi) share ONE catalogId, so the owner label keeps their metadata entries distinct — a + * plain catalogId key would collapse them onto one metadata and misroute. Under a + * {@link org.apache.doris.connector.api.ConnectorStatementScope#NONE NONE} scope (offline / no statement) the + * factory runs on every call — byte-identical to the pre-funnel behavior. Only fe-connector-api types are + * touched, so no fe-core dependency is introduced. The returned metadata and any handle it produces are used + * ONLY through the parent-first SPI interfaces and MUST NOT be cast (the sibling's concrete iceberg/hudi types + * would CCE across the loader split). + */ + private ConnectorMetadata memoizedSiblingMetadata(ConnectorSession session, Connector owner, String ownerLabel) { + String key = "metadata:" + session.getCatalogId() + ":" + ownerLabel; + return session.getStatementScope().getOrCreateMetadata(key, () -> owner.getMetadata(session)); + } + /** * The embedded iceberg sibling's metadata resolved BY TYPE, for the getTableHandle ICEBERG divert only (an - * iceberg-detected table has no handle yet, so the sibling is force-built and asked directly). Obtained fresh - * per call — parity with fe-core, which acquires a ConnectorMetadata per operation; the heavy catalog/caches - * live on the single (memoized) sibling connector, so this is cheap. The returned metadata and any handle it - * produces are used ONLY through the parent-first SPI interfaces and MUST NOT be cast (the sibling's concrete - * iceberg types would CCE across the loader split). + * iceberg-detected table has no handle yet, so the sibling is force-built and asked directly). Routed through + * {@link #memoizedSiblingMetadata} under {@link SiblingOwner#ICEBERG_LABEL}, so the divert and the same + * statement's later per-handle forwards share ONE iceberg metadata instance. The returned metadata and any + * handle it produces are used ONLY through the parent-first SPI interfaces and MUST NOT be cast. * *

    Package-private (not private) so HiveConnectorThreeWayRoutingTest can assert that * {@link HiveConnector#getMetadata} wires the iceberg by-TYPE supplier to THIS arm (the two same-typed * supplier args are otherwise transposable at that sole production wiring point). */ ConnectorMetadata icebergSiblingMetadata(ConnectorSession session) { - return icebergSiblingSupplier.get().getMetadata(session); + return memoizedSiblingMetadata(session, icebergSiblingSupplier.get(), SiblingOwner.ICEBERG_LABEL); } /** * The embedded hudi sibling's metadata resolved BY TYPE, for the getTableHandle HUDI divert only (a * hudi-detected table has no handle yet, so the sibling is force-built and asked directly). Same lifecycle and - * casting contract as {@link #icebergSiblingMetadata}: obtained fresh per call, used ONLY through the - * parent-first SPI interfaces, and never cast (the sibling's concrete hudi types would CCE across the loader - * split). + * casting contract as {@link #icebergSiblingMetadata}: routed through {@link #memoizedSiblingMetadata} under + * {@link SiblingOwner#HUDI_LABEL}, used ONLY through the parent-first SPI interfaces, and never cast. * *

    Package-private (not private) so HiveConnectorThreeWayRoutingTest can assert that * {@link HiveConnector#getMetadata} wires the hudi by-TYPE supplier to THIS arm (see * {@link #icebergSiblingMetadata}). */ ConnectorMetadata hudiSiblingMetadata(ConnectorSession session) { - return hudiSiblingSupplier.get().getMetadata(session); + return memoizedSiblingMetadata(session, hudiSiblingSupplier.get(), SiblingOwner.HUDI_LABEL); } /** * The OWNING sibling's metadata for a foreign (non-hive) table handle, resolved BY HANDLE (3-way ownsHandle - * dispatch over the already-built iceberg / hudi siblings — see HiveConnector.resolveSiblingOwner). Every - * per-handle guard-and-forward method routes through here so a hudi handle reaches the hudi sibling and an - * iceberg handle the iceberg sibling. Obtained fresh per call; the handle is used ONLY through the - * parent-first SPI interfaces and MUST NOT be cast (cross-loader CCE). + * dispatch over the already-built iceberg / hudi siblings — see HiveConnector.resolveSiblingOwnerLabeled). + * Every per-handle guard-and-forward method (and the per-handle {@code beginTransaction} open) routes through + * here, so a hudi handle reaches the hudi sibling and an iceberg handle the iceberg sibling. The resolver + * supplies the owner label from its matched arm, and {@link #memoizedSiblingMetadata} keys the funnel by it — + * so a statement's forwards for one owner share ONE metadata instance, and the by-HANDLE label matches the + * by-TYPE label above (same owner → same key → the getTableHandle divert and these forwards reuse + * one instance). The handle is used ONLY through the parent-first SPI interfaces and MUST NOT be cast + * (cross-loader CCE). */ private ConnectorMetadata siblingMetadata(ConnectorSession session, ConnectorTableHandle handle) { - return siblingOwnerResolver.apply(handle).getMetadata(session); + SiblingOwner owner = siblingOwnerResolver.apply(handle); + return memoizedSiblingMetadata(session, owner.connector(), owner.label()); } // ========== ConnectorSchemaOps ========== @@ -425,9 +448,10 @@ public ConnectorTableSchema getTableSchema( // capabilities govern the table). Only hasScanCapability consumers read the marker, so a capability // that is not per-table-refinable (view / show-create / mvcc) is inert here. Resolve the owner ONCE // (getMetadata is not free) and reuse it for the schema build and the capability read. - Connector owner = siblingOwnerResolver.apply(handle); - ConnectorTableSchema siblingSchema = owner.getMetadata(session).getTableSchema(session, handle); - return reflectSiblingScanCapabilities(owner, siblingSchema); + SiblingOwner owner = siblingOwnerResolver.apply(handle); + ConnectorTableSchema siblingSchema = memoizedSiblingMetadata(session, owner.connector(), owner.label()) + .getTableSchema(session, handle); + return reflectSiblingScanCapabilities(owner.connector(), siblingSchema); } HiveTableHandle hiveHandle = (HiveTableHandle) handle; String dbName = hiveHandle.getDbName(); diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/SiblingOwner.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/SiblingOwner.java new file mode 100644 index 00000000000000..890b5e9b8f712e --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/SiblingOwner.java @@ -0,0 +1,60 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.Connector; + +/** + * The embedded sibling connector that owns a foreign (iceberg/hudi-on-HMS) table, paired with its stable owner + * label. Produced by the 3-way ownsHandle dispatch ({@code HiveConnector.resolveSiblingOwnerLabeled}) so the + * per-handle gateway seams learn WHICH sibling owns a handle AND its label in a single peek (no force-build, no + * identity comparison). + * + *

    The label is the single source of truth for the per-statement metadata funnel key + * ({@code "metadata:" + catalogId + ":" + label}) that lets a statement reuse ONE sibling metadata across all its + * forwards. It MUST be identical on the by-TYPE divert path ({@code getTableHandle}, which has no handle yet and + * asks a sibling by type) and the by-HANDLE forward path (every per-handle guard-and-forward) for the same owner + * — otherwise the two paths would memoize two sibling metadata instances under two keys within one statement. + * Both routes therefore key off {@link #ICEBERG_LABEL} / {@link #HUDI_LABEL} here. + * + *

    The connector is held ONLY as the parent-first {@link Connector} interface — never cast (its concrete + * iceberg/hudi types are invisible across the plugin classloader split; a cast would CCE). + */ +final class SiblingOwner { + + /** Owner label for the embedded iceberg sibling — the funnel-key discriminator, shared by both routes. */ + static final String ICEBERG_LABEL = "iceberg"; + /** Owner label for the embedded hudi sibling — the funnel-key discriminator, shared by both routes. */ + static final String HUDI_LABEL = "hudi"; + + private final Connector connector; + private final String label; + + SiblingOwner(Connector connector, String label) { + this.connector = connector; + this.label = label; + } + + Connector connector() { + return connector; + } + + String label() { + return label; + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java index 91089872bfc484..c0121cb70fbba0 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -24,6 +24,7 @@ import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorTableStatistics; import org.apache.doris.connector.api.ConnectorType; @@ -82,6 +83,12 @@ private static final class ForeignHandle implements ConnectorTableHandle { private final RecordingSiblingMetadata siblingMetadata = new RecordingSiblingMetadata(); private final RecordingSiblingConnector siblingConnector = new RecordingSiblingConnector(siblingMetadata); + // A session whose per-statement scope is NONE (offline): the sibling-metadata funnel runs its factory on every + // forward, so the sibling is consulted per call exactly as before the funnel — these forwarding assertions are + // byte-equivalent to the pre-funnel behavior. Per-statement REUSE (one sibling metadata across many forwards) + // is pinned by the "per-statement sibling-metadata funnel" tests below, which run under a live TestStatementScope. + private final ConnectorSession session = new ScopeSession(1L, "q1", ConnectorStatementScope.NONE); + /** * The by-TYPE force-build supplier constructor arg. This suite exercises only per-handle (by-handle) sites — * which must ALL route via the peek resolver — and never calls getTableHandle (the only by-type site), so the @@ -103,7 +110,8 @@ private static final class ForeignHandle implements ConnectorTableHandle { */ private HiveConnectorMetadata withSibling() { return new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), - SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, handle -> siblingConnector); + SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, + handle -> new SiblingOwner(siblingConnector, SiblingOwner.ICEBERG_LABEL)); } private HiveTableHandle hiveHandle() { @@ -115,34 +123,34 @@ public void everyPerHandleMethodForwardsAForeignHandleToTheSibling() { HiveConnectorMetadata md = withSibling(); // ---- set (a): methods hive overrides — a foreign handle must NOT run hive logic, it must divert ---- - md.getTableSchema(null, foreignHandle); - md.getColumnHandles(null, foreignHandle); - md.getTableStatistics(null, foreignHandle); - md.getColumnStatistics(null, foreignHandle, "c"); - long size = md.estimateDataSizeByListingFiles(null, foreignHandle); - List> timelineRows = md.getMetadataTableRows(null, foreignHandle, "timeline"); - Optional> filter = md.applyFilter(null, foreignHandle, null); - List partNames = md.listPartitionNames(null, foreignHandle); - md.listPartitions(null, foreignHandle, Optional.empty()); - ConnectorMvccSnapshot pin = md.beginQuerySnapshot(null, foreignHandle).orElse(null); - md.getTableFreshness(null, foreignHandle); - md.getPartitionFreshnessMillis(null, foreignHandle, "p"); - md.dropTable(null, foreignHandle); - md.truncateTable(null, foreignHandle, Collections.emptyList()); + md.getTableSchema(session, foreignHandle); + md.getColumnHandles(session, foreignHandle); + md.getTableStatistics(session, foreignHandle); + md.getColumnStatistics(session, foreignHandle, "c"); + long size = md.estimateDataSizeByListingFiles(session, foreignHandle); + List> timelineRows = md.getMetadataTableRows(session, foreignHandle, "timeline"); + Optional> filter = md.applyFilter(session, foreignHandle, null); + List partNames = md.listPartitionNames(session, foreignHandle); + md.listPartitions(session, foreignHandle, Optional.empty()); + ConnectorMvccSnapshot pin = md.beginQuerySnapshot(session, foreignHandle).orElse(null); + md.getTableFreshness(session, foreignHandle); + md.getPartitionFreshnessMillis(session, foreignHandle, "p"); + md.dropTable(session, foreignHandle); + md.truncateTable(session, foreignHandle, Collections.emptyList()); // ---- set (b): methods hive does NOT override — the silent gaps that must be filled by forwarding ---- - md.getTableSchema(null, foreignHandle, null); - md.getMvccPartitionView(null, foreignHandle); - md.resolveTimeTravel(null, foreignHandle, null); - ConnectorTableHandle afterSnapshot = md.applySnapshot(null, foreignHandle, null); - List predicates = md.getSyntheticScanPredicates(null, foreignHandle, null); - ConnectorTableHandle afterScope = md.applyRewriteFileScope(null, foreignHandle, Collections.emptySet()); - ConnectorTableHandle afterTopn = md.applyTopnLazyMaterialization(null, foreignHandle); - List sysTables = md.listSupportedSysTables(null, foreignHandle); - Optional sysHandle = md.getSysTableHandle(null, foreignHandle, "snapshots"); + md.getTableSchema(session, foreignHandle, null); + md.getMvccPartitionView(session, foreignHandle); + md.resolveTimeTravel(session, foreignHandle, null); + ConnectorTableHandle afterSnapshot = md.applySnapshot(session, foreignHandle, null); + List predicates = md.getSyntheticScanPredicates(session, foreignHandle, null); + ConnectorTableHandle afterScope = md.applyRewriteFileScope(session, foreignHandle, Collections.emptySet()); + ConnectorTableHandle afterTopn = md.applyTopnLazyMaterialization(session, foreignHandle); + List sysTables = md.listSupportedSysTables(session, foreignHandle); + Optional sysHandle = md.getSysTableHandle(session, foreignHandle, "snapshots"); // "snapshots" (not "partitions"): hive's own logic returns false for it, so a true answer proves the // reply came from the sibling, not hive. - boolean sysIsTvf = md.isPartitionValuesSysTable(null, foreignHandle, "snapshots"); + boolean sysIsTvf = md.isPartitionValuesSysTable(session, foreignHandle, "snapshots"); // Every per-handle method reached the sibling (proves the divert covers the whole surface). Assertions.assertEquals(RecordingSiblingMetadata.EXPECTED_METHODS, siblingMetadata.calls, @@ -186,24 +194,24 @@ public void hiveHandleRunsHiveBranchAndNeverConsultsSibling() { // The set-(b) + beginQuerySnapshot branches reproduce the SPI default / hive pin WITHOUT the sibling and // without touching the (null) hmsClient — proving the guard falls through to the hive path for a hive handle. - Assertions.assertFalse(md.getMvccPartitionView(null, hive).isPresent(), "hive has no range partition view"); - Assertions.assertFalse(md.resolveTimeTravel(null, hive, null).isPresent(), "hive has no time travel"); - Assertions.assertSame(hive, md.applySnapshot(null, hive, null), "hive applySnapshot returns the handle"); - Assertions.assertTrue(md.getSyntheticScanPredicates(null, hive, null).isEmpty(), + Assertions.assertFalse(md.getMvccPartitionView(session, hive).isPresent(), "hive has no range partition view"); + Assertions.assertFalse(md.resolveTimeTravel(session, hive, null).isPresent(), "hive has no time travel"); + Assertions.assertSame(hive, md.applySnapshot(session, hive, null), "hive applySnapshot returns the handle"); + Assertions.assertTrue(md.getSyntheticScanPredicates(session, hive, null).isEmpty(), "plain hive has no synthetic scan predicate"); - Assertions.assertSame(hive, md.applyRewriteFileScope(null, hive, Collections.emptySet()), + Assertions.assertSame(hive, md.applyRewriteFileScope(session, hive, Collections.emptySet()), "hive applyRewriteFileScope returns the handle"); - Assertions.assertSame(hive, md.applyTopnLazyMaterialization(null, hive), + Assertions.assertSame(hive, md.applyTopnLazyMaterialization(session, hive), "hive applyTopnLazyMaterialization returns the handle"); - Assertions.assertEquals(Collections.singletonList("partitions"), md.listSupportedSysTables(null, hive), + Assertions.assertEquals(Collections.singletonList("partitions"), md.listSupportedSysTables(session, hive), "hive exposes the partitions sys table (t$partitions), served by the partition_values TVF"); - Assertions.assertTrue(md.isPartitionValuesSysTable(null, hive, "partitions"), + Assertions.assertTrue(md.isPartitionValuesSysTable(session, hive, "partitions"), "hive's partitions sys table is TVF-backed"); - Assertions.assertFalse(md.isPartitionValuesSysTable(null, hive, "snapshots"), + Assertions.assertFalse(md.isPartitionValuesSysTable(session, hive, "snapshots"), "hive exposes no sys table other than partitions"); - Assertions.assertFalse(md.getSysTableHandle(null, hive, "snapshots").isPresent(), + Assertions.assertFalse(md.getSysTableHandle(session, hive, "snapshots").isPresent(), "hive's TVF-backed sys table has no native handle"); - ConnectorMvccSnapshot pin = md.beginQuerySnapshot(null, hive).orElse(null); + ConnectorMvccSnapshot pin = md.beginQuerySnapshot(session, hive).orElse(null); Assertions.assertNotNull(pin); Assertions.assertEquals(-1L, pin.getSnapshotId(), "hive's pin is the empty (-1) last-modified pin"); Assertions.assertTrue(pin.isLastModifiedFreshness(), "hive's pin flags last-modified freshness"); @@ -219,7 +227,7 @@ public void foreignHandleFailsLoudWhenNoSiblingConfigured() { // raise a clear error, not NPE deep in a forward. HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext()); - Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableSchema(null, foreignHandle), + Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableSchema(session, foreignHandle), "a foreign handle with no sibling configured must fail loud"); } @@ -230,25 +238,25 @@ public void everyAlterDdlAndValidateMethodForwardsAForeignHandleToTheSibling() { // The 14 ALTER-DDL mutators + 2 write validators: a foreign (iceberg-on-HMS) handle must divert, never // run the hive branch and never be cast. Change objects are null — the guard fires on the handle type // before any param is touched. - md.renameTable(null, foreignHandle, "new"); - md.addColumn(null, foreignHandle, null, null); - md.addColumns(null, foreignHandle, Collections.emptyList()); - md.dropColumn(null, foreignHandle, "c"); - md.renameColumn(null, foreignHandle, "a", "b"); - md.modifyColumn(null, foreignHandle, null, null); - md.reorderColumns(null, foreignHandle, Collections.emptyList()); - md.createOrReplaceBranch(null, foreignHandle, null); - md.createOrReplaceTag(null, foreignHandle, null); - md.dropBranch(null, foreignHandle, null); - md.dropTag(null, foreignHandle, null); - md.addPartitionField(null, foreignHandle, null); - md.dropPartitionField(null, foreignHandle, null); - md.replacePartitionField(null, foreignHandle, null); - md.validateRowLevelDmlMode(null, foreignHandle, null); - md.validateStaticPartitionColumns(null, foreignHandle, Collections.emptyList()); + md.renameTable(session, foreignHandle, "new"); + md.addColumn(session, foreignHandle, null, null); + md.addColumns(session, foreignHandle, Collections.emptyList()); + md.dropColumn(session, foreignHandle, "c"); + md.renameColumn(session, foreignHandle, "a", "b"); + md.modifyColumn(session, foreignHandle, null, null); + md.reorderColumns(session, foreignHandle, Collections.emptyList()); + md.createOrReplaceBranch(session, foreignHandle, null); + md.createOrReplaceTag(session, foreignHandle, null); + md.dropBranch(session, foreignHandle, null); + md.dropTag(session, foreignHandle, null); + md.addPartitionField(session, foreignHandle, null); + md.dropPartitionField(session, foreignHandle, null); + md.replacePartitionField(session, foreignHandle, null); + md.validateRowLevelDmlMode(session, foreignHandle, null); + md.validateStaticPartitionColumns(session, foreignHandle, Collections.emptyList()); // Empty list on purpose: a foreign handle must forward REGARDLESS of emptiness (the empty-early-return is // hive-only) — this would fail if the empty check were placed before the foreign-handle divert. - md.validateWritePartitionNames(null, foreignHandle, Collections.emptyList()); + md.validateWritePartitionNames(session, foreignHandle, Collections.emptyList()); Assertions.assertEquals(RecordingSiblingMetadata.EXPECTED_WRITE_METHODS, siblingMetadata.calls, "every ALTER-DDL mutator + write validator must forward a foreign handle to the sibling"); @@ -263,12 +271,12 @@ public void hiveHandleRejectsNonEmptyPartitionNamesWithLegacyMessage() { // partition-NAME list form INSERT ... PARTITION(p1, p2) is unsupported on a hive table. UNLIKE the two // permissive validators, a hive handle here THROWS the EXACT legacy message on a non-empty list. The e2e // test_hive_write_type.groovy asserts on this literal substring, so it must stay byte-identical. - assertThrowsMessage(() -> md.validateWritePartitionNames(null, hive, Arrays.asList("p1", "p2")), + assertThrowsMessage(() -> md.validateWritePartitionNames(session, hive, Arrays.asList("p1", "p2")), "Not support insert with partition spec in hive catalog."); // An empty list (a plain INSERT ... SELECT or a static PARTITION(col='val') INSERT) is legal plain-hive // and MUST return silently — a throw here would newly reject legal writes. - md.validateWritePartitionNames(null, hive, Collections.emptyList()); + md.validateWritePartitionNames(session, hive, Collections.emptyList()); Assertions.assertEquals(0, siblingConnector.getMetadataCount, "a hive handle must never build/consult the iceberg sibling to validate partition names"); @@ -282,26 +290,26 @@ public void hiveHandleAlterDdlThrowsAndValidateIsNoopAndNeverConsultsSibling() { // Group-1: ALTER-DDL for a hive handle throws the EXACT inherited SPI-default message (byte-parity with // pre-override behavior) without building or consulting the sibling. - assertThrowsMessage(() -> md.renameTable(null, hive, "n"), "RENAME TABLE not supported"); - assertThrowsMessage(() -> md.addColumn(null, hive, null, null), "ADD COLUMN not supported"); - assertThrowsMessage(() -> md.addColumns(null, hive, Collections.emptyList()), "ADD COLUMNS not supported"); - assertThrowsMessage(() -> md.dropColumn(null, hive, "c"), "DROP COLUMN not supported"); - assertThrowsMessage(() -> md.renameColumn(null, hive, "a", "b"), "RENAME COLUMN not supported"); - assertThrowsMessage(() -> md.modifyColumn(null, hive, null, null), "MODIFY COLUMN not supported"); - assertThrowsMessage(() -> md.reorderColumns(null, hive, Collections.emptyList()), + assertThrowsMessage(() -> md.renameTable(session, hive, "n"), "RENAME TABLE not supported"); + assertThrowsMessage(() -> md.addColumn(session, hive, null, null), "ADD COLUMN not supported"); + assertThrowsMessage(() -> md.addColumns(session, hive, Collections.emptyList()), "ADD COLUMNS not supported"); + assertThrowsMessage(() -> md.dropColumn(session, hive, "c"), "DROP COLUMN not supported"); + assertThrowsMessage(() -> md.renameColumn(session, hive, "a", "b"), "RENAME COLUMN not supported"); + assertThrowsMessage(() -> md.modifyColumn(session, hive, null, null), "MODIFY COLUMN not supported"); + assertThrowsMessage(() -> md.reorderColumns(session, hive, Collections.emptyList()), "REORDER COLUMNS not supported"); - assertThrowsMessage(() -> md.createOrReplaceBranch(null, hive, null), "CREATE/REPLACE BRANCH not supported"); - assertThrowsMessage(() -> md.createOrReplaceTag(null, hive, null), "CREATE/REPLACE TAG not supported"); - assertThrowsMessage(() -> md.dropBranch(null, hive, null), "DROP BRANCH not supported"); - assertThrowsMessage(() -> md.dropTag(null, hive, null), "DROP TAG not supported"); - assertThrowsMessage(() -> md.addPartitionField(null, hive, null), "ADD PARTITION FIELD not supported"); - assertThrowsMessage(() -> md.dropPartitionField(null, hive, null), "DROP PARTITION FIELD not supported"); - assertThrowsMessage(() -> md.replacePartitionField(null, hive, null), "REPLACE PARTITION FIELD not supported"); + assertThrowsMessage(() -> md.createOrReplaceBranch(session, hive, null), "CREATE/REPLACE BRANCH not supported"); + assertThrowsMessage(() -> md.createOrReplaceTag(session, hive, null), "CREATE/REPLACE TAG not supported"); + assertThrowsMessage(() -> md.dropBranch(session, hive, null), "DROP BRANCH not supported"); + assertThrowsMessage(() -> md.dropTag(session, hive, null), "DROP TAG not supported"); + assertThrowsMessage(() -> md.addPartitionField(session, hive, null), "ADD PARTITION FIELD not supported"); + assertThrowsMessage(() -> md.dropPartitionField(session, hive, null), "DROP PARTITION FIELD not supported"); + assertThrowsMessage(() -> md.replacePartitionField(session, hive, null), "REPLACE PARTITION FIELD not supported"); // Group-2: validate* for a hive handle MUST return silently — a throw here would newly reject legal // plain-hive row-level DML / static-partition INSERTs. - md.validateRowLevelDmlMode(null, hive, null); - md.validateStaticPartitionColumns(null, hive, Collections.emptyList()); + md.validateRowLevelDmlMode(session, hive, null); + md.validateStaticPartitionColumns(session, hive, Collections.emptyList()); Assertions.assertEquals(0, siblingConnector.getMetadataCount, "a hive handle must never build/consult the iceberg sibling for ALTER-DDL / validate"); @@ -315,7 +323,7 @@ public void beginTransactionForwardsAForeignHandleToTheSibling() { // A foreign (iceberg-on-HMS) write must open the SIBLING's transaction, so iceberg's write plan can // downcast the session-bound transaction to IcebergConnectorTransaction — a HiveConnectorTransaction // (what the unconditional open would bind) would ClassCastException there. - ConnectorTransaction txn = md.beginTransaction(null, foreignHandle); + ConnectorTransaction txn = md.beginTransaction(session, foreignHandle); Assertions.assertSame(RecordingSiblingMetadata.SIBLING_TXN, txn, "a foreign handle must open the sibling's transaction, not a hive one"); @@ -332,14 +340,15 @@ public void beginTransactionForHiveHandleOpensHiveTxnAndNeverConsultsSibling() { // sibling. The selection must be symmetric — hive and iceberg write plans downcast to different types. ConnectorTransaction hiveTxn = new NoOpConnectorTransaction(70099L, "HIVE"); HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), - SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, handle -> siblingConnector) { + SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, + handle -> new SiblingOwner(siblingConnector, SiblingOwner.ICEBERG_LABEL)) { @Override public ConnectorTransaction beginTransaction(ConnectorSession session) { return hiveTxn; } }; - Assertions.assertSame(hiveTxn, md.beginTransaction(null, hiveHandle()), + Assertions.assertSame(hiveTxn, md.beginTransaction(session, hiveHandle()), "a hive handle must open the connector-level (hive) transaction, not the sibling's"); Assertions.assertEquals(0, siblingConnector.getMetadataCount, "a hive handle must never build/consult the iceberg sibling to open a transaction"); @@ -360,9 +369,10 @@ public void foreignHandleSchemaReflectsSiblingScanCapabilitiesAsPerTableMarker() ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE); HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, - handle -> new CapabilityDeclaringSiblingConnector(siblingCaps)); + handle -> new SiblingOwner(new CapabilityDeclaringSiblingConnector(siblingCaps), + SiblingOwner.ICEBERG_LABEL)); - ConnectorTableSchema schema = md.getTableSchema(null, foreignHandle); + ConnectorTableSchema schema = md.getTableSchema(session, foreignHandle); String csv = schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY); Assertions.assertNotNull(csv, "the delegated schema must carry the reflected per-table capability marker"); List names = Arrays.asList(csv.split(",")); @@ -382,7 +392,7 @@ public void foreignHandleSchemaUnchangedWhenSiblingDeclaresNoCapabilities() { // lacks auto-analyze) is pinned by foreignHandleSchemaWithholdsAutoAnalyzeFromRealHudiSibling below. // MUTATION: dropping the isEmpty() early-return and stamping an (empty) marker unconditionally -> red here. HiveConnectorMetadata md = withSibling(); // RecordingSiblingConnector declares no capabilities - ConnectorTableSchema schema = md.getTableSchema(null, foreignHandle); + ConnectorTableSchema schema = md.getTableSchema(session, foreignHandle); Assertions.assertNull(schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY), "no marker when the sibling declares no capabilities"); } @@ -399,10 +409,10 @@ public void foreignHandleSchemaWithholdsAutoAnalyzeFromRealHudiSibling() { // flag) -> the marker would contain SUPPORTS_COLUMN_AUTO_ANALYZE -> red here. HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, - handle -> new CapabilityDeclaringSiblingConnector( - EnumSet.of(ConnectorCapability.SUPPORTS_METADATA_TABLE))); + handle -> new SiblingOwner(new CapabilityDeclaringSiblingConnector( + EnumSet.of(ConnectorCapability.SUPPORTS_METADATA_TABLE)), SiblingOwner.ICEBERG_LABEL)); - ConnectorTableSchema schema = md.getTableSchema(null, foreignHandle); + ConnectorTableSchema schema = md.getTableSchema(session, foreignHandle); String csv = schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY); Assertions.assertNotNull(csv, "a non-empty hudi sibling must still stamp its declared capabilities"); List names = Arrays.asList(csv.split(",")); @@ -416,6 +426,102 @@ public void foreignHandleSchemaWithholdsAutoAnalyzeFromRealHudiSibling() { "hudi-on-HMS gains no nested-column prune from a sibling that does not declare it"); } + // ============== per-statement sibling-metadata funnel (HMS heterogeneous gateway) ============== + // Within one statement, the gateway obtains ONE sibling ConnectorMetadata per (catalogId, owner) and reuses it + // across every forward (read / scan / DDL / MVCC / the write-transaction open), keyed + // "metadata::" on the session's per-statement scope — mirroring fe-core's own funnel for + // a plain connector. RecordingSiblingConnector.getMetadataCount is the load count. + + @Test + public void liveScopeSharesOneSiblingMetadataAcrossEveryForwardIncludingTheWriteTxn() { + // Many read forwards (including the getTableSchema stray) + the per-handle beginTransaction open, all in one + // statement -> the sibling is built ONCE and every forward (reads AND the write transaction) reuses it. + // MUTATION: not memoizing -> a build per forward -> count > 1 -> red. + HiveConnectorMetadata md = withSibling(); + ConnectorSession live = new ScopeSession(1L, "q1", new TestStatementScope()); + + md.getColumnHandles(live, foreignHandle); + md.listPartitionNames(live, foreignHandle); + md.getMetadataTableRows(live, foreignHandle, "timeline"); + md.getTableSchema(live, foreignHandle); + md.beginTransaction(live, foreignHandle); + + Assertions.assertEquals(1, siblingConnector.getMetadataCount, + "one sibling metadata per (catalog, owner) per statement — reads and the write txn share it"); + } + + @Test + public void noneScopeBuildsAFreshSiblingMetadataEachForward() { + // No live statement scope (offline / no ConnectContext): the funnel factory runs on every forward, exactly + // as before the funnel existed. This is the byte-equivalence guard for the NONE path. + HiveConnectorMetadata md = withSibling(); + + md.getColumnHandles(session, foreignHandle); + md.listPartitionNames(session, foreignHandle); + md.getColumnHandles(session, foreignHandle); + + Assertions.assertEquals(3, siblingConnector.getMetadataCount, + "NONE scope -> the sibling metadata factory runs on every forward (pre-funnel behavior)"); + } + + @Test + public void byTypeDivertAndByHandleForwardShareOneSiblingMetadata() { + // The getTableHandle divert asks the sibling BY TYPE (icebergSiblingMetadata) before any handle exists; the + // later per-handle forwards resolve BY HANDLE. Both must mint the SAME funnel key for the same owner (the + // by-TYPE literal label == the by-HANDLE resolver-arm label), or the statement would hold two sibling + // metadata instances. Wire the iceberg by-TYPE supplier to the SAME recording connector the resolver + // returns, then drive both paths under one live scope. MUTATION: mismatched labels -> two builds -> red. + HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), + () -> siblingConnector, SUPPLIER_MUST_NOT_BE_USED, + handle -> new SiblingOwner(siblingConnector, SiblingOwner.ICEBERG_LABEL)); + ConnectorSession live = new ScopeSession(1L, "q1", new TestStatementScope()); + + md.icebergSiblingMetadata(live); // by-TYPE (getTableHandle divert path) + md.getColumnHandles(live, foreignHandle); // by-HANDLE (per-handle forward) + + Assertions.assertEquals(1, siblingConnector.getMetadataCount, + "the by-TYPE divert and the by-HANDLE forward key the same owner -> one shared sibling metadata"); + } + + @Test + public void differentCatalogIdIsolatesTheSiblingMetadata() { + // A statement joining two heterogeneous HMS catalogs shares one scope map; the catalog id in the key keeps + // each catalog's sibling metadata isolated. MUTATION: dropping catalogId from the key -> the two collide. + HiveConnectorMetadata md = withSibling(); + TestStatementScope scope = new TestStatementScope(); + + md.getColumnHandles(new ScopeSession(1L, "q1", scope), foreignHandle); + md.getColumnHandles(new ScopeSession(2L, "q1", scope), foreignHandle); + + Assertions.assertEquals(2, siblingConnector.getMetadataCount, + "a different catalog id keys a different sibling metadata (no cross-catalog collapse)"); + } + + @Test + public void icebergAndHudiSiblingsAreIsolatedWithinAStatement() { + // The whole point of the owner label: iceberg-on-HMS and hudi-on-HMS tables of ONE gateway share a catalog + // id, so ONLY the label ("iceberg" vs "hudi") keeps their metadata entries apart. Each owner is built once + // and never collapses onto the other. MUTATION: a shared (label-less) key -> one owner's metadata serves + // the other -> a count is 0 while the other is 2 -> red. + RecordingSiblingConnector hudiConnector = new RecordingSiblingConnector(new RecordingSiblingMetadata()); + ForeignHandle hudiHandle = new ForeignHandle(); + HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), + SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, + handle -> handle == hudiHandle + ? new SiblingOwner(hudiConnector, SiblingOwner.HUDI_LABEL) + : new SiblingOwner(siblingConnector, SiblingOwner.ICEBERG_LABEL)); + ConnectorSession live = new ScopeSession(1L, "q1", new TestStatementScope()); + + md.getColumnHandles(live, foreignHandle); // iceberg owner + md.getColumnHandles(live, hudiHandle); // hudi owner + md.getColumnHandles(live, foreignHandle); // iceberg owner again -> reuse + + Assertions.assertEquals(1, siblingConnector.getMetadataCount, + "the iceberg owner is built once under its label"); + Assertions.assertEquals(1, hudiConnector.getMetadataCount, + "the hudi owner is isolated under its own label and built once (never collapsed onto iceberg)"); + } + private static void assertThrowsMessage(Executable exec, String expectedMessage) { DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, exec); Assertions.assertEquals(expectedMessage, e.getMessage(), diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java index a8996a9ca9c20c..a7aa4830fb39b2 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java @@ -20,6 +20,7 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.hms.HmsClient; @@ -68,10 +69,15 @@ public class HiveConnectorMetadataTableHandleDivertTest { // getTableHandle routes BY TYPE (the two by-TYPE suppliers), NEVER via the by-handle owner resolver — so wire // the owner resolver to fail loud if anything reaches for it. - private static final Function OWNER_RESOLVER_UNUSED = handle -> { + private static final Function OWNER_RESOLVER_UNUSED = handle -> { throw new AssertionError("getTableHandle must divert BY TYPE, never via the by-handle owner resolver"); }; + // getTableHandle's by-TYPE divert routes through the per-statement sibling-metadata funnel, which reads + // session.getStatementScope() + getCatalogId(); a NONE scope makes the factory run every call (byte-equivalent + // to the pre-funnel divert), so these by-type routing assertions are unchanged. + private final ConnectorSession session = new ScopeSession(1L, "q1", ConnectorStatementScope.NONE); + /** The foreign (non-hive) handle a sibling's getTableHandle produces post-flip. */ private static final class ForeignHandle implements ConnectorTableHandle { } @@ -85,7 +91,7 @@ private static final class ForeignHandle implements ConnectorTableHandle { public void icebergTableDivertsToIcebergSiblingNotHudi() { HiveConnectorMetadata md = withSiblings(icebergTable()); - Optional handle = md.getTableHandle(null, "db", "t"); + Optional handle = md.getTableHandle(session, "db", "t"); Assertions.assertTrue(handle.isPresent(), "an existing iceberg-on-HMS table must resolve a handle"); Assertions.assertSame(icebergHandle, handle.get(), @@ -106,7 +112,7 @@ public void icebergDivertPropagatesSiblingEmpty() { icebergSibling.metadata.returnHandle = null; HiveConnectorMetadata md = withSiblings(icebergTable()); - Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(), + Assertions.assertFalse(md.getTableHandle(session, "db", "t").isPresent(), "an empty from the sibling must pass through unchanged"); // Prove the empty was FORWARDED from the sibling (its getTableHandle was consulted), not short-circuited // to empty by the gateway — otherwise a broken `if (ICEBERG) return Optional.empty()` would pass this test. @@ -120,7 +126,7 @@ public void hudiTableDivertsToHudiSiblingNotIceberg() { // OWN foreign handle verbatim — NOT a HiveTableHandle stamped HUDI, and NOT routed to the iceberg sibling. HiveConnectorMetadata md = withSiblings(hiveTable(HUDI)); - Optional handle = md.getTableHandle(null, "db", "t"); + Optional handle = md.getTableHandle(session, "db", "t"); Assertions.assertTrue(handle.isPresent(), "an existing hudi-on-HMS table must resolve a handle"); Assertions.assertSame(hudiHandle, handle.get(), @@ -141,7 +147,7 @@ public void hudiDivertPropagatesSiblingEmpty() { hudiSibling.metadata.returnHandle = null; HiveConnectorMetadata md = withSiblings(hiveTable(HUDI)); - Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(), + Assertions.assertFalse(md.getTableHandle(session, "db", "t").isPresent(), "an empty from the hudi sibling must pass through unchanged"); Assertions.assertEquals(1, hudiSibling.metadata.getTableHandleCalls, "the hudi sibling is authoritative for hudi existence: its getTableHandle must be the source of empty"); @@ -151,7 +157,7 @@ public void hudiDivertPropagatesSiblingEmpty() { public void hiveTableBuildsHiveHandleWithoutConsultingSibling() { HiveConnectorMetadata md = withSiblings(hiveTable(PARQUET)); - Optional handle = md.getTableHandle(null, "db", "t"); + Optional handle = md.getTableHandle(session, "db", "t"); Assertions.assertTrue(handle.get() instanceof HiveTableHandle, "a hive table resolves a HiveTableHandle"); Assertions.assertEquals(HiveTableType.HIVE, ((HiveTableHandle) handle.get()).getTableType()); @@ -167,7 +173,7 @@ public void unknownNonViewTableStillFailsLoud() { // non-view table is still rejected (not swallowed into a hudi divert). HiveConnectorMetadata md = withSiblings(hiveTable(UNKNOWN_FORMAT)); - Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(null, "db", "t"), + Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(session, "db", "t"), "an unsupported non-view input format must fail loud, not be diverted to a sibling"); Assertions.assertEquals(0, hudiSibling.getMetadataCalls, "the UNKNOWN fail-loud must not consult the hudi sibling"); @@ -181,7 +187,7 @@ public void missingTableReturnsEmptyWithoutConsultingSibling() { new FakeHmsClient(icebergTable(), false), Collections.emptyMap(), new FakeConnectorContext(), () -> icebergSibling, () -> hudiSibling, OWNER_RESOLVER_UNUSED); - Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(), + Assertions.assertFalse(md.getTableHandle(session, "db", "t").isPresent(), "a non-existent table short-circuits to empty before any format detection or divert"); Assertions.assertEquals(0, icebergSibling.getMetadataCalls, "a missing table must not build the iceberg sibling"); Assertions.assertEquals(0, hudiSibling.getMetadataCalls, "a missing table must not build the hudi sibling"); @@ -194,7 +200,7 @@ public void icebergTableFailsLoudWhenNoSiblingConfigured() { HiveConnectorMetadata md = new HiveConnectorMetadata( new FakeHmsClient(icebergTable(), true), Collections.emptyMap(), new FakeConnectorContext()); - Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(null, "db", "t"), + Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(session, "db", "t"), "an iceberg table with no sibling configured must fail loud"); } @@ -205,7 +211,7 @@ public void hudiTableFailsLoudWhenNoSiblingConfigured() { HiveConnectorMetadata md = new HiveConnectorMetadata( new FakeHmsClient(hiveTable(HUDI), true), Collections.emptyMap(), new FakeConnectorContext()); - Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(null, "db", "t"), + Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(session, "db", "t"), "a hudi table with no sibling configured must fail loud"); } diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java index e127cac59229c2..e8a85c1830a416 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java @@ -20,6 +20,7 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; @@ -192,15 +193,19 @@ public void getMetadataWiresEachByTypeSupplierToItsOwnSibling() { HiveConnector connector = new HiveConnector(props(), ctx); HiveConnectorMetadata md = connector.newMetadata(null); + // A NONE-scope session: the by-TYPE helpers now route through the per-statement metadata funnel (which + // reads session.getStatementScope()/getCatalogId()), so a null session would NPE after the force-build we + // assert. NONE makes the funnel run its factory straight through — the force-build order is unchanged. + ConnectorSession session = new ScopeSession(1L, "q1", ConnectorStatementScope.NONE); // The getTableHandle ICEBERG arm resolves BY TYPE via icebergSiblingMetadata, which force-builds the // iceberg sibling (createSiblingConnector("iceberg")). A transposed getMetadata would build hudi here. - md.icebergSiblingMetadata(null); + md.icebergSiblingMetadata(session); Assertions.assertEquals(1, ctx.icebergBuilds, "the iceberg by-TYPE arm must force-build the iceberg sibling"); Assertions.assertEquals(0, ctx.hudiBuilds, "the iceberg by-TYPE arm must NOT build the hudi sibling"); // Symmetrically the HUDI arm must resolve the hudi sibling, not rebuild iceberg. - md.hudiSiblingMetadata(null); + md.hudiSiblingMetadata(session); Assertions.assertEquals(1, ctx.hudiBuilds, "the hudi by-TYPE arm must force-build the hudi sibling"); Assertions.assertEquals(1, ctx.icebergBuilds, "the hudi by-TYPE arm must not rebuild the iceberg sibling"); } diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/ScopeSession.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/ScopeSession.java new file mode 100644 index 00000000000000..f2e1d6d7c29ff8 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/ScopeSession.java @@ -0,0 +1,89 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorStatementScope; + +import java.util.Collections; +import java.util.Map; + +/** + * Minimal {@link ConnectorSession} for hive-connector unit tests, carrying a catalog id and a per-statement + * scope. The heterogeneous gateway keys its per-statement sibling-metadata funnel on + * {@code "metadata:" + getCatalogId() + ":" + ownerLabel} and reads the scope via {@link #getStatementScope()}, + * so a test wires a live {@link TestStatementScope} to prove sharing (or {@link ConnectorStatementScope#NONE} to + * prove the pre-funnel load-every-time behavior). Mirrors the iceberg connector's test session. + */ +final class ScopeSession implements ConnectorSession { + + private final long catalogId; + private final String queryId; + private final ConnectorStatementScope scope; + + ScopeSession(long catalogId, String queryId, ConnectorStatementScope scope) { + this.catalogId = catalogId; + this.queryId = queryId; + this.scope = scope; + } + + @Override + public long getCatalogId() { + return catalogId; + } + + @Override + public String getQueryId() { + return queryId; + } + + @Override + public ConnectorStatementScope getStatementScope() { + return scope; + } + + @Override + public String getUser() { + return "u"; + } + + @Override + public String getTimeZone() { + return "UTC"; + } + + @Override + public String getLocale() { + return "en_US"; + } + + @Override + public String getCatalogName() { + return "c"; + } + + @Override + public T getProperty(String name, Class type) { + return null; + } + + @Override + public Map getCatalogProperties() { + return Collections.emptyMap(); + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/TestStatementScope.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/TestStatementScope.java new file mode 100644 index 00000000000000..7d52c162a12dac --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/TestStatementScope.java @@ -0,0 +1,40 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorStatementScope; + +import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Supplier; + +/** + * A memoizing {@link ConnectorStatementScope} for hive-connector unit tests. The hive connector module does not + * depend on fe-core, so tests cannot use the engine's {@code ConnectorStatementScopeImpl}; this is a faithful + * copy of it (mirrors the iceberg connector's test copy). Sharing one instance across a statement's forwards lets + * a test prove that the heterogeneous gateway reuses ONE sibling metadata per owner per statement. + */ +final class TestStatementScope implements ConnectorStatementScope { + + private final ConcurrentHashMap cache = new ConcurrentHashMap<>(); + + @Override + @SuppressWarnings("unchecked") + public T computeIfAbsent(String key, Supplier loader) { + return (T) cache.computeIfAbsent(key, k -> loader.get()); + } +} From fe1aecfae4b3d265a1eabedb3e50286ee8bf97b7 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 22:22:56 +0800 Subject: [PATCH 37/47] [doc](catalog) per-statement metadata: STEP 2 (HMS sibling funnel) closeout + STEP 3 handoff Record the HMS heterogeneous-gateway sibling-metadata funnel landing (commit 5fd55d0a32a): grounding correction (4 acquisition sites, not ~43), the design confirmed with the user, e2e deferred to the unified delegation pass, 348 tests green + gates green + adversarial review clean. Flip RD-2 to done in tasklist and rewrite HANDOFF to point at STEP 3 (write-side sharing: reroute the 8 write seams through the funnel, uphold the hive start-of-write refresh + read/write identity gates, lift ConnectorTransaction ownership). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../per-statement-table-owner-port/HANDOFF.md | 75 ++++++++++--------- .../progress.md | 9 +++ .../tasklist.md | 2 +- 3 files changed, 48 insertions(+), 38 deletions(-) diff --git a/plan-doc/per-statement-table-owner-port/HANDOFF.md b/plan-doc/per-statement-table-owner-port/HANDOFF.md index 9f6042f1f63c88..967c7c695c07ab 100644 --- a/plan-doc/per-statement-table-owner-port/HANDOFF.md +++ b/plan-doc/per-statement-table-owner-port/HANDOFF.md @@ -5,59 +5,60 @@ --- -# 🆕 本轮（2026-07-19 session 5）已完成：**防漂移门禁落地 → 读取键石（STEP 1 / RD-1）完全收官** +# 🆕 本轮（2026-07-19 session 6）已完成：**HMS 异构网关兄弟元数据每语句去重（读取键石之后的第二大步）完全落地** ## 一句话结果 -- 用户拍板方案 A（现在就上门禁锁死读取侧、写入 8 处显式豁免）。 -- **已实现（全绿，已提交 `b2d147998d1`）**： - - 新增 `tools/check-fecore-metadata-funnel.sh`（bash grep 门禁，仿 `check-connector-imports.sh`）+ 自测 `.test.sh`：扫 `fe/fe-core/src/main/java`，禁裸 `Connector#getMetadata(session)`；放行=①funnel 文件 `PluginDrivenMetadata.java`（含 javadoc）②带 `getMetadata-funnel-exempt` 标记的行（call 行**或其上一行**）③无参 `getMetadata()`（异方法）④注释行。正则双形（同行参数 + 换行参数），不误伤 `getMetadataTableRows`/API 定义。 - - 8 处写入裸调各加一行上置标记注释（103 字）；挂入 `fe/fe-core/pom.xml` validate 阶段 exec（与 fe-connector 门禁同深度同范式）。 -- **验证**：自测 PASS（10 项含核心/换行/白名单/同行+上行标记/无参/边界/注释/退出码/标记承重）；门禁真实树 exit 0、8 处未标记 exit 1（都证过）；fe-core checkstyle 0 违规；`mvn -pl fe-core validate` 实跑触发 exec + BUILD SUCCESS。 -- **读取键石收官**：C1 地基 + C2 关闭 + C3 改道 + 扫描存字段 + 后台读穿 + 防漂移门禁全落地（RD-1 = ✅）。 +- **背景**：一个 HMS 目录同时管 Hive/Iceberg/Hudi 三类表，Hive 连接器充当网关、把 Iceberg/Hudi 表的操作转发给内嵌"兄弟连接器"。此前网关**每转发一次就新建一个兄弟元数据外壳**（一条 SELECT 扫一张 HMS-Iceberg 表要重建十几次）。本轮把兄弟元数据也纳入"每语句一实例"。 +- **grounding 纠偏**：整个 hive 连接器"取兄弟元数据"仅 **4 处**（3 个 helper + `getTableSchema` 旁路）；文档"~43 处 per-handle 改道"实为误导——40+ 处转发 + `beginTransaction(session,handle)` 全穿第三个 helper、改动零行。 +- **已落地（全绿，commit `5fd55d0a32a`）**： + - 新增 `SiblingOwner{connector,label}`（`ICEBERG_LABEL`/`HUDI_LABEL` 常量单一真源）；`HiveConnector.resolveSiblingOwnerLabeled` 命中臂带标签、`resolveSiblingOwner` 委派它（3 个 provider seam 字节不变）。 + - `HiveConnectorMetadata.memoizedSiblingMetadata` key=`metadata::

    This object's own job is the deterministic statement-end backstop. {@link #finalizeAtStatementEnd()} rolls + * back a transaction the executor never committed or rolled back — only a mid-flight abort leaves one active — + * and the scope runs it BEFORE closing the shared metadata, so a transaction is always finished before the + * instance it was minted from is closed. It is idempotent and can never undo a committed write: on every normal + * path the executor has already finished the transaction, which removes it from the manager, so the backstop + * finds nothing active and does nothing.

    + * + *

    fe-core-internal and connector-agnostic: it traffics only in the neutral {@link ConnectorWriteOps} / + * {@link ConnectorSession} / {@link ConnectorTransaction} SPI types (never a concrete connector type) and is + * stored on the connector-agnostic {@link org.apache.doris.connector.api.ConnectorStatementScope} as an opaque + * value, recognized by {@link org.apache.doris.connector.ConnectorStatementScopeImpl} only to order its + * teardown before metadata close.

    + */ +public final class CatalogStatementTransaction { + + private static final Logger LOG = LogManager.getLogger(CatalogStatementTransaction.class); + + /** Sentinel for "no transaction opened yet" (never a real connector txn id). */ + public static final long INVALID_TXN_ID = -1L; + + // The write facet of the statement's one shared metadata instance; the transaction is minted from it. Held + // for co-hold coherence — the metadata itself is closed by the scope's generic pass, this class only + // finalizes the transaction. + private final ConnectorWriteOps writeOps; + private final ConnectorSession session; + private final PluginDrivenTransactionManager transactionManager; + + private ConnectorTransaction connectorTx; + private long txnId = INVALID_TXN_ID; + + public CatalogStatementTransaction(ConnectorWriteOps writeOps, ConnectorSession session, + PluginDrivenTransactionManager transactionManager) { + this.writeOps = writeOps; + this.session = session; + this.transactionManager = transactionManager; + } + + /** + * Opens the write transaction from the shared metadata and registers it with the manager (which also + * publishes it in the global registry the BE block-allocation RPC / commit-data feedback look up by id), + * returning it so the executor can bind it onto the sink session. Called once, from the insert executor's + * {@code beginTransaction}. + */ + public ConnectorTransaction begin(ConnectorTableHandle writeHandle) { + connectorTx = writeOps.beginTransaction(session, writeHandle); + txnId = transactionManager.begin(connectorTx); + return connectorTx; + } + + public long getTransactionId() { + return txnId; + } + + public ConnectorTransaction getConnectorTransaction() { + return connectorTx; + } + + /** + * Statement-end backstop: if the transaction is still active (the executor never reached commit / rollback, + * i.e. the statement was aborted mid-flight), roll it back. On every normal path the executor already + * finished it — the manager no longer holds it — so this is a no-op and can never undo a committed write. + * The scope runs this before it closes the shared metadata instance. + */ + public void finalizeAtStatementEnd() { + if (txnId == INVALID_TXN_ID || !transactionManager.isActive(txnId)) { + return; + } + LOG.warn("Statement ended with an uncommitted plugin-driven transaction {}; rolling it back as a backstop", + txnId); + transactionManager.rollback(txnId); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java index 373223fb980f0c..ab3265c18927f7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java @@ -25,6 +25,7 @@ import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.ConnectorTransaction; import org.apache.doris.datasource.ExternalTable; +import org.apache.doris.datasource.plugin.CatalogStatementTransaction; import org.apache.doris.datasource.plugin.PluginDrivenExternalCatalog; import org.apache.doris.datasource.plugin.PluginDrivenExternalTable; import org.apache.doris.datasource.plugin.PluginDrivenMetadata; @@ -87,8 +88,19 @@ public void beginTransaction() { // downcasts; a single-format connector ignores the handle (the SPI default delegates to the no-arg // beginTransaction). resolveWriteTargetHandle fails loud rather than handing the gateway a null handle. ConnectorTableHandle writeHandle = ((PluginDrivenExternalTable) table).resolveWriteTargetHandle(); - connectorTx = writeOps.beginTransaction(connectorSession, writeHandle); - txnId = ((PluginDrivenTransactionManager) transactionManager).begin(connectorTx); + // Co-hold the transaction with the statement's one shared metadata (writeOps) + session on the statement + // scope, so read and write share one instance and the scope deterministically rolls back a transaction + // aborted mid-flight at statement end -- before it closes that shared metadata. begin() still mints from + // writeOps and registers with the manager (global lookup for the BE block-allocation RPC / commit-data + // feedback), returning the transaction for the sink-session binding in finalizeSink. Under NONE the scope + // stores nothing, so the holder is transient and the executor's own commit/rollback remains the only + // lifecycle (byte-identical to the pre-co-holder path). + CatalogStatementTransaction stmtTxn = (CatalogStatementTransaction) connectorSession.getStatementScope() + .computeIfAbsent("txn:" + connectorSession.getCatalogId(), + () -> new CatalogStatementTransaction(writeOps, connectorSession, + (PluginDrivenTransactionManager) transactionManager)); + connectorTx = stmtTxn.begin(writeHandle); + txnId = connectorTx.getTransactionId(); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java index bb65871c88b16a..16c230f8d7de40 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/transaction/PluginDrivenTransactionManager.java @@ -114,6 +114,16 @@ public Transaction getTransaction(long id) throws UserException { return txn; } + /** + * Whether transaction {@code id} is still open here, i.e. registered by {@link #begin(ConnectorTransaction)} + * and not yet committed or rolled back (both {@link #commit(long)} and {@link #rollback(long)} remove it). + * The statement scope's end-of-statement backstop uses this to tell a mid-flight-aborted transaction from + * one the executor already finished, so it rolls back only genuine orphans and never a committed write. + */ + public boolean isActive(long id) { + return transactions.containsKey(id); + } + /** * Internal transaction record. When {@code connectorTx} is non-null (every plugin-driven * write) the SPI is the source of truth and commit/rollback delegate to it; close() always diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java index 14d11d6d0462d0..b7d868e4a3f43e 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorStatementScopeTest.java @@ -17,14 +17,23 @@ package org.apache.doris.connector; +import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorStatementScope; +import org.apache.doris.connector.api.ConnectorWriteOps; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.datasource.plugin.CatalogStatementTransaction; import org.apache.doris.nereids.StatementContext; import org.apache.doris.qe.ConnectContext; +import org.apache.doris.transaction.PluginDrivenTransactionManager; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import org.mockito.Mockito; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; import java.util.concurrent.atomic.AtomicInteger; /** @@ -115,6 +124,38 @@ public void closeAllClosesCloseableValuesOnceAndIgnoresPlainValues() throws Exce "each closeable value is closed exactly once across repeated closeAll (idempotent)"); } + @Test + public void closeAllFinalizesTransactionsBeforeClosingMetadata() { + // Two-pass teardown: the scope must finalize (roll back an orphaned) write transaction BEFORE it closes + // the shared metadata instance the transaction was minted from, so the transaction is never left holding a + // closed instance. MUTATION: single-pass close (or closing metadata first) -> the recorded order flips + // -> red. + List order = new ArrayList<>(); + PluginDrivenTransactionManager mgr = new PluginDrivenTransactionManager(); + ConnectorTransaction tx = Mockito.mock(ConnectorTransaction.class); + Mockito.when(tx.getTransactionId()).thenReturn(90001L); + Mockito.doAnswer(inv -> { + order.add("txn-rollback"); + return null; + }).when(tx).rollback(); + ConnectorWriteOps ops = Mockito.mock(ConnectorWriteOps.class); + Mockito.when(ops.beginTransaction(Mockito.any(), Mockito.any())).thenReturn(tx); + + ConnectorStatementScopeImpl scope = new ConnectorStatementScopeImpl(); + // The statement's shared metadata: a closeable that records the moment it is closed. + AutoCloseable metadata = () -> order.add("metadata-close"); + scope.computeIfAbsent("metadata:1", () -> metadata); + CatalogStatementTransaction holder = + new CatalogStatementTransaction(ops, Mockito.mock(ConnectorSession.class), mgr); + scope.computeIfAbsent("txn:1", () -> holder); + holder.begin(new ConnectorTableHandle() { }); // active orphan: the executor never committed it + + scope.closeAll(); + + Assertions.assertEquals(Arrays.asList("txn-rollback", "metadata-close"), order, + "the transaction is finalized before the shared metadata is closed"); + } + @Test public void resetClosesTheDroppedScopeBeforeStartingFresh() { // A prepared EXECUTE / retry reuses one StatementContext and calls resetConnectorStatementScope() at the diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/CatalogStatementTransactionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/CatalogStatementTransactionTest.java new file mode 100644 index 00000000000000..385f43c2caced9 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/plugin/CatalogStatementTransactionTest.java @@ -0,0 +1,128 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.plugin; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorWriteOps; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.transaction.PluginDrivenTransactionManager; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +/** + * Pins {@link CatalogStatementTransaction}, the per-statement owner of a plugin-driven write transaction: + * {@link CatalogStatementTransaction#begin} mints the transaction from the statement's shared write-ops and + * registers it with the manager, and the statement-end backstop + * ({@link CatalogStatementTransaction#finalizeAtStatementEnd}) rolls back only a genuinely orphaned + * transaction (mid-flight abort) and NEVER undoes one the executor already committed or rolled back. + */ +public class CatalogStatementTransactionTest { + + private static ConnectorTransaction tx(long id) { + ConnectorTransaction tx = Mockito.mock(ConnectorTransaction.class); + Mockito.when(tx.getTransactionId()).thenReturn(id); + return tx; + } + + private static ConnectorWriteOps writeOpsReturning(ConnectorTransaction tx) { + ConnectorWriteOps ops = Mockito.mock(ConnectorWriteOps.class); + Mockito.when(ops.beginTransaction(Mockito.any(), Mockito.any())).thenReturn(tx); + return ops; + } + + private static CatalogStatementTransaction holder(ConnectorTransaction tx, PluginDrivenTransactionManager mgr) { + return new CatalogStatementTransaction(writeOpsReturning(tx), Mockito.mock(ConnectorSession.class), mgr); + } + + @Test + public void beginMintsFromWriteOpsAndRegistersWithManager() { + // begin() opens the transaction from the statement's ONE shared metadata (writeOps) and registers it, so + // the write inherits the read arm's client/ops and the BE RPC can look it up by id. MUTATION: not + // registering -> isActive false -> red. + PluginDrivenTransactionManager mgr = new PluginDrivenTransactionManager(); + ConnectorTransaction tx = tx(80001L); + CatalogStatementTransaction holder = holder(tx, mgr); + + ConnectorTransaction opened = holder.begin(new ConnectorTableHandle() { }); + + Assertions.assertSame(tx, opened, "begin returns the transaction minted from the shared write-ops"); + Assertions.assertEquals(80001L, holder.getTransactionId(), "the holder stamps the connector txn id"); + Assertions.assertTrue(mgr.isActive(80001L), "the transaction is registered active with the manager"); + } + + @Test + public void finalizeRollsBackAnOrphanedTransaction() { + // A statement aborted mid-flight leaves the transaction active (the executor reached neither commit nor + // rollback). The statement-end backstop rolls it back. MUTATION: skipping the rollback -> the orphan + // leaks its resources -> this verify fails. + PluginDrivenTransactionManager mgr = new PluginDrivenTransactionManager(); + ConnectorTransaction tx = tx(80002L); + CatalogStatementTransaction holder = holder(tx, mgr); + holder.begin(new ConnectorTableHandle() { }); + + holder.finalizeAtStatementEnd(); + + Mockito.verify(tx).rollback(); + Assertions.assertFalse(mgr.isActive(80002L), "the orphan is deregistered after the backstop rollback"); + } + + @Test + public void finalizeNeverUndoesACommittedTransaction() throws Exception { + // THE safety property: on the normal path the executor commits (removing the txn from the manager), so + // the statement-end backstop must find nothing active and NOT roll back -- otherwise it would undo a + // durably committed write. MUTATION: dropping the isActive guard -> finalize rolls back a committed txn. + PluginDrivenTransactionManager mgr = new PluginDrivenTransactionManager(); + ConnectorTransaction tx = tx(80003L); + CatalogStatementTransaction holder = holder(tx, mgr); + holder.begin(new ConnectorTableHandle() { }); + mgr.commit(80003L); // the executor's onComplete path finished it + + holder.finalizeAtStatementEnd(); + + Mockito.verify(tx).commit(); + Mockito.verify(tx, Mockito.never()).rollback(); + } + + @Test + public void finalizeIsANoOpAfterRollback() throws Exception { + // The executor's onFail already rolled back; the backstop must be idempotent -- roll back once, not + // twice. MUTATION: dropping the isActive guard -> a second rollback -> red. + PluginDrivenTransactionManager mgr = new PluginDrivenTransactionManager(); + ConnectorTransaction tx = tx(80004L); + CatalogStatementTransaction holder = holder(tx, mgr); + holder.begin(new ConnectorTableHandle() { }); + mgr.rollback(80004L); // the executor's onFail path finished it + + holder.finalizeAtStatementEnd(); + + Mockito.verify(tx, Mockito.times(1)).rollback(); + } + + @Test + public void finalizeIsANoOpWhenNoTransactionWasEverOpened() { + // The empty-insert path never calls begin(); a stray finalize must not touch the manager or NPE. + PluginDrivenTransactionManager mgr = new PluginDrivenTransactionManager(); + CatalogStatementTransaction holder = holder(tx(80005L), mgr); + + Assertions.assertEquals(CatalogStatementTransaction.INVALID_TXN_ID, holder.getTransactionId()); + Assertions.assertDoesNotThrow(holder::finalizeAtStatementEnd); + } +} From e1db78e83743670caeeb3ac7071aab2d2764e5db Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 19 Jul 2026 23:41:13 +0800 Subject: [PATCH 40/47] [doc](catalog) per-statement metadata: write-sharing closeout (RD-3) + STEP 4 handoff Record the read/write-shares-one-metadata-instance step as done: the P3 implementation design + as-built (3a reroute + identity gate, 3b co-holder + two-pass closeAll, side-car orthogonality), the RD-3 tasklist row (two commits), the session-7 progress entry, and a rewritten HANDOFF pointing at the cache isolation step (STEP 4, the independent security track: getIdentityShardKey() SPI, iceberg projection-cache key sharding, fe-core schema-cache bypass, the heterogeneous-gateway hole, anti-drift gate, and the required threat-model sign-off). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3 --- .../per-statement-table-owner-port/HANDOFF.md | 80 +++++++++---------- .../P3-write-sharing-implementation-design.md | 56 +++++++++++++ .../progress.md | 10 +++ .../tasklist.md | 4 +- 4 files changed, 105 insertions(+), 45 deletions(-) create mode 100644 plan-doc/per-statement-table-owner-port/designs/P3-write-sharing-implementation-design.md diff --git a/plan-doc/per-statement-table-owner-port/HANDOFF.md b/plan-doc/per-statement-table-owner-port/HANDOFF.md index 967c7c695c07ab..de9a4de0004299 100644 --- a/plan-doc/per-statement-table-owner-port/HANDOFF.md +++ b/plan-doc/per-statement-table-owner-port/HANDOFF.md @@ -5,60 +5,54 @@ --- -# 🆕 本轮（2026-07-19 session 6）已完成：**HMS 异构网关兄弟元数据每语句去重（读取键石之后的第二大步）完全落地** +# 🆕 本轮（2026-07-19 session 7）已完成：**写入共用一个每语句元数据实例 + 事务归属上移（读写共用步骤完全落地）** ## 一句话结果 -- **背景**：一个 HMS 目录同时管 Hive/Iceberg/Hudi 三类表，Hive 连接器充当网关、把 Iceberg/Hudi 表的操作转发给内嵌"兄弟连接器"。此前网关**每转发一次就新建一个兄弟元数据外壳**（一条 SELECT 扫一张 HMS-Iceberg 表要重建十几次）。本轮把兄弟元数据也纳入"每语句一实例"。 -- **grounding 纠偏**：整个 hive 连接器"取兄弟元数据"仅 **4 处**（3 个 helper + `getTableSchema` 旁路）；文档"~43 处 per-handle 改道"实为误导——40+ 处转发 + `beginTransaction(session,handle)` 全穿第三个 helper、改动零行。 -- **已落地（全绿，commit `5fd55d0a32a`）**： - - 新增 `SiblingOwner{connector,label}`（`ICEBERG_LABEL`/`HUDI_LABEL` 常量单一真源）；`HiveConnector.resolveSiblingOwnerLabeled` 命中臂带标签、`resolveSiblingOwner` 委派它（3 个 provider seam 字节不变）。 - - `HiveConnectorMetadata.memoizedSiblingMetadata` key=`metadata::