Skip to content

[Feature] Add ADBC external catalog support - #78714

Open
metegenez wants to merge 32 commits into
StarRocks:mainfrom
metegenez:feature/adbc-catalog-mvp
Open

metegenez wants to merge 32 commits into
StarRocks:mainfrom
metegenez:feature/adbc-catalog-mvp

Conversation

@metegenez

@metegenez metegenez commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Why I'm doing

Apache Arrow ADBC is evolving quickly, with a growing driver ecosystem. I expect broader adoption across OLAP engines in the near term. Its Arrow batch interface reduces row-by-row conversion overhead and can make bulk analytical reads substantially faster than JDBC's row-oriented API (Java performance comparison).

This is the first step toward broader ADBC support: a small, read-only integration through the ADBC Driver Manager, with Flight SQL as the first validated source. If we agree on this direction, I plan to follow up with broader driver coverage, source-specific metadata support and additional capabilities.

For background, I previously explored ADBC in #70297, where the Go/CGO dependency prompted the Driver Manager suggestion. I waited for ADBC 24 to complete the FE integration using upstream JNI metadata APIs. This connector builds without Go/CGO and loads source drivers at runtime.

What I'm doing

Fixes #69868

Add a read-only adbc external catalog. ADBC db_schema maps to a StarRocks database. Scans support projection, predicate and LIMIT pushdown through one stream.

  • No bundled source drivers: StarRocks packages the Driver Manager and FE JNI bridge, but no database driver, including adbc_driver_flightsql. Users install their chosen compatible driver on each FE and BE/CN.
  • Designed for more drivers: source-specific metadata mapping is the main extension point; compatible drivers reuse the same scan path. New sources still need SQL/type compatibility validation.
flowchart LR
    FE["FE: metadata + JNI + Driver Manager"] -->|scan plan| BE["BE/CN: scan + Driver Manager"]
    FE --> FD["User-installed driver on FE"]
    BE --> BD["User-installed driver on BE/CN"]
    FD --> Source["Data source: Flight SQL verified"]
    BD -->|"SQL / Arrow batches"| Source
Loading

Uses ADBC Java 0.24.0 with Arrow Java 18.0.0. Writes and distributed partitions are outside this version.

Diff size: 5,550 added lines: 2,426 implementation/build, 2,529 tests, and 595 Markdown docs/guides. There are 18 deleted lines.

Catalog creation

CREATE EXTERNAL CATALOG flight_sql PROPERTIES (
    "type" = "adbc",
    "driver" = "adbc_driver_flightsql",
    "uri" = "grpc+tls://flight.example.com:31337",
    "username" = "<user>",
    "password" = "<password>"
);

SHOW DATABASES FROM flight_sql;
SELECT n_nationkey, n_name FROM flight_sql.main.nation
WHERE n_nationkey = 24 LIMIT 1;

See the ADBC catalog guide for driver installation and FE configuration. EN/ZH/JA docs are included. The AI-generated Japanese translation needs native-speaker review.

Validation

Check Result
Upstream CI Passed: Thirdparty Ubuntu/Rocky, BUILD, FE/BE UT, SQL-Tester, ADMIT and restart checks
Upstream incremental coverage FE 91.48%; BE 95.94%
FE ADBC unit tests + Checkstyle 95 passed; 0 style violations
Neighboring FE tests Passed: JDBC, Fluss, EXPLAIN, SHOW CREATE, partition traits and MV rewrite
BE ADBC scanner + descriptor tests (ASAN) 20 passed
BE factory regressions 2 passed against production libraries
Production FE/BE build Passed (ARM64 Docker)
Clean ADBC + JNI dependency build Passed with pinned Maven 3.9.11
Live Flight SQL E2E Passed (SQLFlite 1.5.5 / DuckDB 1.1.1)
BE module boundaries / Thrift compatibility / clang-format-10 Passed

Live E2E was run manually; it is not yet in the SQL integration suite. Coverage: metadata, pushdown, quoted columns/literals, scalar values, 10,000 rows across chunks, DDL redaction and execution profiles.

Test environment

I verified the builds on my ARM64 Docker host with the optional benchmark connector and StarCache disabled. SQLFlite's embedded-quote table-name limitation has unit and direct JNI coverage.

Observability: ADBC uses catalog_type="adbc" for FE query metrics and BE scan metrics. Existing scan profiles, driver/metadata error context and credential masking are preserved. ADBC scans render in optimizer plans and query history. MV freshness checks retain the existing refresh logs, timers and query-cache counters.

What type of PR is this

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: new adbc catalog, metadata and scan support
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: ADBC native build and JNI packaging

Checklist

  • I have added test cases for my new feature.
  • This PR needs user documentation.
    • I have added documentation for my new feature.
    • This PR needs auto-generated documentation.
  • This is a backport PR.

Bugfix cherry-pick branch check

  • I have checked the version labels which the PR must be backported to.
    • 4.1
    • 4.0
    • 3.5

Not applicable: new feature, no backport requested.

@github-actions

github-actions Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

New parameters without documentation

This PR introduces 5 parameter(s) not yet documented. Please add entries before or shortly after merge.

BE Configuration — 5 undocumented
Parameter Type Default Mutable Description Suggested doc
enable_put_combinded_txn_log_parallel Bool false Yes — docs/en/administration/configuration/BE_parameters/
lake_pk_index_block_cache_limit_percent Int32 10 No — docs/en/administration/configuration/BE_parameters/
llm_cache_size Int32 131072 No — docs/en/administration/configuration/BE_parameters/
llm_max_concurrent_queries Int32 8 No — docs/en/administration/configuration/BE_parameters/
llm_max_queue_size Int32 4096 No Legacy ai_query runtime configuration. It is intentionally independent from the AI function runtime. docs/en/administration/configuration/BE_parameters/

Auto-generated by build-support/extract_and_diff_params.py. See docs/en/administration/configuration/ for the doc files to update.

@CelerData-Reviewer

Copy link
Copy Markdown

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-07T08:57:51.473370Z 6b23a98 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: becbe235ea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread be/src/exec/exec_factory.cpp Resolved
@CelerData-Reviewer

Copy link
Copy Markdown

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ddacb1c446

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@CelerData-Reviewer

Copy link
Copy Markdown

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c323599e73

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread be/src/exec/adbc_scanner.cpp Resolved
@CelerData-Reviewer

Copy link
Copy Markdown

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eb7797e35d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread be/src/exec/adbc_scanner.cpp Resolved
Comment thread be/src/exec/adbc_scanner.cpp Resolved
@CelerData-Reviewer

Copy link
Copy Markdown

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ece20340c3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@metegenez

Copy link
Copy Markdown
Contributor Author

Re-run the workflow

@CelerData-Reviewer

Copy link
Copy Markdown

@codex review

@metegenez

Copy link
Copy Markdown
Contributor Author

Re-run the workflow

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6b23a98d8e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@metegenez

metegenez commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor Author

@kevincai Hey Kevin, looks like the CI runners are having trouble reaching GitHub. Thirdparty Update keeps timing out on github.com:443 before it even checks out the PR code. I retried it a few times, but it keeps failing the same way, and the Re-run the workflow command is failing too. I can’t really get the PR CI moving because of this. Could you take a look when you have a chance? Latest one: https://github.com/StarRocks/starrocks/actions/runs/34106425242/job/101694403123

Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
@metegenez
metegenez force-pushed the feature/adbc-catalog-mvp branch from 7765590 to 769b184 Compare October 16, 2026 10:31
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

🌎 Translation Required?

✅ All translation files are up to date.
Great job! No translation actions are required for this PR.

🕒 Last updated: Tue, 27 Oct 2026 15:53:28 GMT

Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
Merge current main without rewriting the PR history. Point the English,
Chinese and Japanese configuration templates to existing FE/BE reference
pages and use local system-variable links.

Validate with Lychee 0.23.0: 7651 local links passed; 139 targeted FE tests
and FE build passed. Schema compatibility, BE module boundaries and
handbook checks passed. Documentation-only fix; existing ADBC profiles,
logs and metrics are preserved.

Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
@tokoko

tokoko commented Oct 27, 2026

Copy link
Copy Markdown

I tested locally with postgres. Mostly worked fine except when driver returns opaque extension types like extension<arrow.opaque[storage_type=string, type_name=numeric, vendor_name=PostgreSQL]> which connector fails on.

Also I think eventually it would be better to handle pushdown sql generation for adbc seperately when apache/arrow#49796 is merged and drivers self-declare relevant info codes.

@metegenez

metegenez commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor Author

I tested locally with postgres. Mostly worked fine except when driver returns opaque extension types like extension<arrow.opaque[storage_type=string, type_name=numeric, vendor_name=PostgreSQL]> which connector fails on.

Also I think eventually it would be better to handle pushdown sql generation for adbc seperately when apache/arrow#49796 is merged and drivers self-declare relevant info codes.

PostgreSQL driver does that, the decimal/numeric problem. Starrocks arrow converter doesnt have a layer for opaque extension, that is why I try to push "flightsql-only" narrative for now, otherwise supporting every ADBC driver would take time and investment on arrow side of starrocks.

ADBC feature can make lots of improvements on how Starrocks interract with modern engines, in agentic era, including pushdown, as you said.

Signed-off-by: Metehan Yildirim <metegenez@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

@github-actions

Copy link
Copy Markdown
Contributor

[FE Incremental Coverage Report]

✅ pass : 450 / 493 (91.28%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/catalog/ADBCPartitionKey.java 0 3 00.00% [25, 26, 30]
🔵 com/starrocks/server/GlobalStateMgr.java 0 2 00.00% [3572, 3573]
🔵 com/starrocks/sql/optimizer/statistics/StatisticsCalculator.java 1 2 50.00% [1101]
🔵 com/starrocks/connector/adbc/ADBCTableName.java 13 19 68.42% [35, 39, 43, 48, 55, 58]
🔵 com/starrocks/catalog/Table.java 3 4 75.00% [705]
🔵 com/starrocks/connector/partitiontraits/ADBCPartitionTraits.java 7 9 77.78% [34, 73]
🔵 com/starrocks/connector/adbc/ADBCSchemaResolver.java 58 70 82.86% [74, 75, 76, 81, 82, 83, 125, 126, 140, 141, 151, 152]
🔵 com/starrocks/planner/ADBCScanNode.java 65 75 86.67% [67, 73, 127, 135, 140, 141, 178, 183, 184, 188]
🔵 com/starrocks/catalog/ADBCTable.java 41 45 91.11% [37, 38, 87, 133]
🔵 com/starrocks/connector/adbc/ADBCMetadata.java 95 97 97.94% [132, 202]
🔵 com/starrocks/sql/optimizer/transformer/RelationTransformer.java 3 3 100.00% []
🔵 com/starrocks/connector/adbc/ADBCConnector.java 69 69 100.00% []
🔵 com/starrocks/sql/optimizer/operator/physical/PhysicalADBCScanOperator.java 4 4 100.00% []
🔵 com/starrocks/sql/analyzer/AstToStringBuilder.java 11 11 100.00% []
🔵 com/starrocks/common/Config.java 1 1 100.00% []
🔵 com/starrocks/sql/optimizer/operator/OperatorVisitor.java 2 2 100.00% []
🔵 com/starrocks/qe/StmtExecutor.java 1 1 100.00% []
🔵 com/starrocks/sql/optimizer/rule/transformation/MergeLimitDirectRule.java 1 1 100.00% []
🔵 com/starrocks/sql/optimizer/rule/RuleType.java 1 1 100.00% []
🔵 com/starrocks/connector/ConnectorType.java 1 1 100.00% []
🔵 com/starrocks/sql/optimizer/operator/pattern/MultiOpPattern.java 1 1 100.00% []
🔵 com/starrocks/sql/optimizer/LogicalPlanPrinter.java 1 1 100.00% []
🔵 com/starrocks/sql/optimizer/operator/OperatorType.java 2 2 100.00% []
🔵 com/starrocks/sql/optimizer/rule/implementation/ADBCScanImplementationRule.java 6 6 100.00% []
🔵 com/starrocks/sql/plan/PlanFragmentBuilder.java 31 31 100.00% []
🔵 com/starrocks/sql/optimizer/dump/DesensitizedSQLBuilder.java 7 7 100.00% []
🔵 com/starrocks/sql/Explain.java 11 11 100.00% []
🔵 com/starrocks/sql/optimizer/OutputPropertyDeriver.java 1 1 100.00% []
🔵 com/starrocks/connector/partitiontraits/CachedPartitionTraits.java 3 3 100.00% []
🔵 com/starrocks/sql/optimizer/OptExpressionVisitor.java 1 1 100.00% []
🔵 com/starrocks/connector/ConnectorPartitionTraits.java 1 1 100.00% []
🔵 com/starrocks/sql/optimizer/operator/logical/LogicalADBCScanOperator.java 8 8 100.00% []

@github-actions

Copy link
Copy Markdown
Contributor

[BE Incremental Coverage Report]

✅ pass : 307 / 320 (95.94%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 be/src/exec/adbc_scanner.cpp 196 209 93.78% [94, 95, 169, 220, 256, 260, 261, 275, 276, 334, 335, 344, 345]
🔵 be/src/connector/adbc_connector.cpp 80 80 100.00% []
🔵 be/src/exec/exec_factory.cpp 6 6 100.00% []
🔵 be/src/runtime/descriptors_ext.h 1 1 100.00% []
🔵 be/src/exec/adbc_scanner.h 1 1 100.00% []
🔵 be/src/runtime/descriptors_ext.cpp 12 12 100.00% []
🔵 be/src/exec_primitive/exec_node.cpp 1 1 100.00% []
🔵 be/src/runtime/descriptors.cpp 2 2 100.00% []
🔵 be/src/module/connector_bootstrap.cpp 1 1 100.00% []
🔵 be/src/connector/adbc_connector.h 7 7 100.00% []

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[New Feature] Native ADBC External Catalog

4 participants