[feature] (cloud) Add Alibaba Cloud OSS C++ SDK stack - Part 1

[nsivarajan]

What problem does this PR solve?

Issue Number: close #65793
Related PR: part 1 of 4 — see issue for full series

PR's to follow : PR2: #65794

Problem Summary

Doris BE currently accesses OSS through the AWS S3-compatible SDK path. This works for basic AK/SK operations but cannot use ECS instance profile credentials, STS AssumeRole, RRSA, or environment variable providers — all of which require the native Alibaba Cloud OSS C++ SDK.

The fe-filesystem-oss SPI module only supports static AK/SK credentials for OSS. Users running Doris on Alibaba Cloud ECS must embed long-lived access keys in configuration even when an ECS RAM role is attached, and cannot use STS AssumeRole or Kubernetes RRSA (RAM Roles for Service Accounts) for pod-level identity.

This PR completes the FE-side credential chain for the fe-filesystem-oss SPI, adding type-based dispatch via oss.credentials_provider and support for:

• ECS instance profile — fetches temporary credentials from the ECS metadata service; no AK/SK required when a RAM role is attached to the instance
• STS AssumeRole — cross-account and least-privilege access via oss.role_arn / oss.external_id
• RRSA (OIDC) — Kubernetes pod identity via OIDC token file (oss.oidc_token_file + oss.oidc_provider_arn)
• Environment variables — ALIBABA_CLOUD_ACCESS_KEY_ID / SECRET
• Default chain — tries env → config file → ECS metadata in order

This is PR 1 of 4. It is additive and does not change any existing code paths. The native OSS C++ SDK (BE-side routing) is wired in PR-3, gated behind enable_oss_native_sdk = false by default.

---

Changes

fe/fe-filesystem/fe-filesystem-oss/

• pom.xml — add aliyun-java-sdk-sts 3.1.3 and aliyun-java-sdk-core 4.7.6
• AliyunCredentialsBridge.java (new) — adapts AlibabaCloudCredentialsProvider to the OSS SDK's CredentialsProvider interface; handles session token extraction for BasicSessionCredentials
• OssCredentialsProviderType.java (new) — enum for oss.credentials_provider values with aliases (ecs_ram_role → INSTANCE_PROFILE, web_identity → OIDC)
• OssFileSystemProperties.java — toBackendKv() now includes "provider"="OSS" so BE can route to the native SDK path
• OssObjStorage.java — buildOssClient() switches on OssCredentialsProviderType; ECS role name auto-discovered from metadata if oss.ecs_ram_role_name unset

fe/fe-core/ (additive only)

• OSSProperties.java — adds roleArn (OSS_ROLE_ARN), externalId (AWS_EXTERNAL_ID); getBackendConfigProperties() now emits "provider"="OSS", AWS_ROLE_ARN, AWS_EXTERNAL_ID for BE credential routing

---

Release note

Add oss.credentials_provider type-based dispatch for ECSinstance profile, STS AssumeRole, RRSA, and environment variable credential modes. Existing static AK/SK path is unchanged. All changes are additive.

---

Check List
Test:

• Unit test — OssCredentialsProviderTypeTest (alias mapping, enum parsing)
• Manual test — requires live OSS bucket with ECS RAM role (deferred to PR-3)
• Note: BE-side routing (enable_oss_native_sdk) is in PR-3; this PR is pure FE logic

Behavior changed:

• Yes — new oss.credentials_provider property key accepted in OSS catalog/SPI
  properties. No existing property keys changed. Default behavior (static AK/SK)
  is preserved when oss.credentials_provider is absent.

Documentation:

• Yes — OSS catalog credential configuration guide (to be added with PR-3)

[Thearas]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[nsivarajan]

run buildall

[dataroaring]

/review

[github-actions]

Code Review: OSS Native Storage Vault Support

This PR adds a significant new feature — native Alibaba Cloud OSS storage vault support with ECS instance profile, STS AssumeRole, and explicit AK/SK authentication. The architecture closely mirrors the existing S3 implementation, which is a good structural choice. However, the review identified several critical and high-severity issues that must be addressed before merge.

---

Critical Checkpoint Conclusions (per code-review skill Part 1.3)

1. Does the code accomplish its goal?

• Partially. The code implements the OSS file system, reader, writer, credential providers, cloud recycler accessor, and FE vault management. However, several correctness bugs and missing pieces prevent it from being production-ready.

2. Is the modification small, clear, and focused?

• The scope is large but well-organized across clear module boundaries. However, the change is large enough that thorough testing is essential, and the test coverage has significant gaps.

3. Concurrency:

• Issue found: OSSFileWriter::_complete_part_task_callback writes _st without holding _completed_lock, while the S3 equivalent holds the lock. This is a data race.
• Issue found: OSSFileWriter destructor does not wait on _async_close_pack->future unlike S3, creating use-after-free risk with async close.
• OSSClientHolder concurrency model is acceptable (lock-and-copy pattern for get()).

4. Lifecycle management:

• Issue found: OSS SDK InitializeSdk() is called but ShutdownSdk() is never called (both in BE and cloud modules).
• Critical: OSSFileWriter destructor does not wait on async close future — potential use-after-free.

5. Configuration items:

• New configs max_oss_client_retry, oss_read_base_wait_time_ms, oss_read_max_wait_time_ms are mutable and properly declared.

6. Incompatible changes:

• None. The protobuf enum OSS = 0 already existed. USE_OSS ifdef ensures backward compatibility.

7. Parallel code paths:

• Multiple divergences from S3: Missing check_after_upload verification, missing SCOPED_CONCURRENCY_COUNT, missing compile_check_begin.h, different error handling in exists_impl, different list_impl semantics.

8. Test coverage:

• Critical gap: OSSStorageVaultTest has a bug in validProps() (missing type key) — ALL tests will throw IllegalArgumentException from the base class.
• BE credential provider tests require live cloud infrastructure — no mocked unit tests.
• Regression test lacks negative test cases and does not use order_qt pattern.
• Cloud accessor tests will fail due to missing empty-path guards in delete_directory and list_directory.

9. Observability:

• bvar metrics are added for reader and writer. OSS accessor has latency recorders. Adequate.
• Minor: OSS reuses S3 DorisMetrics (with TODO comments), conflating monitoring.

10. Transaction/persistence:

• CRITICAL: OSSResource is NOT registered in GsonUtils.resourceTypeAdapterFactory. When FE journals an OSSResource creation/alter to EditLog and then replays it on restart, deserialization will fail because the subtype is unknown. This can prevent FE from starting. File: fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java — add .registerSubtype(OSSResource.class, OSSResource.class.getSimpleName()).

11. FE-BE variable passing:

• OSSProperties.getObjStoreInfoPB() does not set use_path_style unlike S3Properties.

12. Performance:

• batch_upload_impl is sequential (no parallelism) — acceptable for now but suboptimal.
• download_impl buffers entire file in memory via rdbuf() — could OOM for large files.

13. Other issues:

• SECURITY: runtime.setIgnoreSSL(true) in STS AssumeRole path disables TLS certificate verification for credential exchange.
• Data safety: parse_oss_last_modified returns 0 (epoch) on parse failure, causing the recycler to treat unparseable objects as maximally old and delete them.
• build.sh never passes BUILD_OSS to cmake for BE — the DISABLE_BUILD_OSS env var has no effect.
• STS AssumeRole always uses ecs_ram_role for base credentials, ignoring user-provided AK/SK.

---

Summary of Issues by Severity

Severity	Count	Key Issues
CRITICAL	3	OSSResource not in GsonUtils (FE crash on restart), setIgnoreSSL(true) in STS path, OSSStorageVaultTest broken
HIGH	8	Writer _st data race, writer destructor use-after-free, delete_directory/list_directory missing guards, exists() swallows errors, parse failure → epoch deletion, STS ignores AK/SK, BUILD_OSS not passed to cmake, OSSResource weak validation
MEDIUM	10+	Missing compile_check, missing check_after_upload, list_impl semantics, retry only handles 2 error codes, etc.
LOW	6+	Metric reuse, temp file naming, jitter range, etc.

Inline comments are attached to the most important issues.

[gavinchou]

Nice PR, however, there may be something need to consider more carefully.

Introducing OSSStorageVault will break the consistency with the existing S3StorageVault which also can access OSS. And, it may introduce extra complexity.
Try to use the provider field and some configurations to make the existing s3 vaults work with native OSS client too. e.g.

if (vault.provider == "OSS" && Config.enable_oss_native_sdk) {
    objclient = new OssRemote(); // check DefaultRemote.java and RemoteBase.java for more info
}

the above pattern should also work for the BE side.

I left some initial feedback on this PR, and feel free to discuss or suggest changes.

[nsivarajan]

Thanks @gavinchou for the suggestion. Agreed on consistency — the modified approach removes OSSStorageVault and reuses S3StorageVault. OSS is detected by endpoint via OSSProperties.guessIsMe() and serialized through OSSProperties.getObjStoreInfoPB(), which stamps provider=OSS in the proto.

On the BE side, provider=OSS + enable_oss_native_sdk routes to the native Alibaba OSS SDK (required for ECS instance profile, STS AssumeRole, RRSA/OIDC); otherwise falls back to the existing AWS SDK S3-compatible path. All other providers are unchanged.

The Thrift storage policy path (PushStoragePolicyTask) and the flat-map path (TVF/Export/Load) are also designed to carry provider=OSS through to the BE so routing is consistent across all paths.

Hope this aligns with the Doris pattern for standardisation and future provider additions — please let me know if any adjustment is needed.

[nsivarajan]

run buildall

[hello-stephen]

TPC-H: Total hot run time: 29376 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 25e389847049906c14d3d6bbf80f5c7ed876b390, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17630	3990	4011	3990
q2	2005	325	199	199
q3	10281	1401	825	825
q4	4680	473	342	342
q5	7506	833	573	573
q6	189	172	147	147
q7	781	812	614	614
q8	9922	1585	1522	1522
q9	6071	4320	4366	4320
q10	6790	1741	1461	1461
q11	508	349	329	329
q12	750	592	453	453
q13	18086	3276	2675	2675
q14	270	260	245	245
q15	q16	794	767	710	710
q17	983	937	997	937
q18	6962	5740	5578	5578
q19	1311	1239	1129	1129
q20	801	696	544	544
q21	5849	2626	2486	2486
q22	442	367	297	297
Total cold run time: 102611 ms
Total hot run time: 29376 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4448	4334	4305	4305
q2	300	321	219	219
q3	4541	4932	4350	4350
q4	2081	2185	1382	1382
q5	4397	4254	4294	4254
q6	232	187	132	132
q7	1731	1967	1749	1749
q8	2500	2197	2094	2094
q9	7745	7777	7813	7777
q10	4673	4652	4194	4194
q11	579	452	401	401
q12	749	755	556	556
q13	3402	3572	2918	2918
q14	331	318	296	296
q15	q16	715	737	653	653
q17	1413	1344	1381	1344
q18	8129	7297	7013	7013
q19	1103	1060	1074	1060
q20	2219	2195	1947	1947
q21	5329	4616	4477	4477
q22	509	457	405	405
Total cold run time: 57126 ms
Total hot run time: 51526 ms

[hello-stephen]

TPC-DS: Total hot run time: 177743 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 25e389847049906c14d3d6bbf80f5c7ed876b390, data reload: false

query5	4317	627	484	484
query6	471	243	217	217
query7	4848	595	358	358
query8	328	195	177	177
query9	8774	4092	4120	4092
query10	501	363	293	293
query11	5923	2345	2137	2137
query12	160	103	102	102
query13	1276	625	447	447
query14	6254	5240	4872	4872
query14_1	4201	4250	4172	4172
query15	207	205	177	177
query16	997	449	444	444
query17	913	697	572	572
query18	2444	454	345	345
query19	206	186	149	149
query20	108	105	105	105
query21	229	158	131	131
query22	13429	13528	13388	13388
query23	17181	16469	16121	16121
query23_1	16194	16457	16528	16457
query24	7460	1728	1264	1264
query24_1	1311	1292	1299	1292
query25	542	433	358	358
query26	1384	353	210	210
query27	2671	621	385	385
query28	4478	1970	2008	1970
query29	1056	600	494	494
query30	339	262	229	229
query31	1115	1099	979	979
query32	111	66	63	63
query33	523	326	262	262
query34	1160	1151	645	645
query35	760	794	673	673
query36	1195	1198	1046	1046
query37	157	107	94	94
query38	1871	1705	1671	1671
query39	878	883	894	883
query39_1	854	840	837	837
query40	262	169	148	148
query41	74	71	72	71
query42	97	99	99	99
query43	331	329	279	279
query44	1420	780	759	759
query45	192	186	179	179
query46	1101	1198	716	716
query47	2193	2153	2040	2040
query48	422	434	304	304
query49	621	421	322	322
query50	1072	442	344	344
query51	10804	10560	10268	10268
query52	86	88	74	74
query53	268	281	208	208
query54	304	238	228	228
query55	77	75	67	67
query56	312	323	303	303
query57	1347	1303	1233	1233
query58	299	268	271	268
query59	1587	1639	1455	1455
query60	313	275	249	249
query61	147	153	146	146
query62	573	495	449	449
query63	242	199	197	197
query64	2817	1062	871	871
query65	4767	4656	4637	4637
query66	1848	512	384	384
query67	29135	29227	29251	29227
query68	3157	1469	967	967
query69	416	305	259	259
query70	1056	950	933	933
query71	364	334	317	317
query72	3058	2564	2377	2377
query73	824	752	450	450
query74	5071	4948	4702	4702
query75	2544	2510	2169	2169
query76	2313	1167	793	793
query77	350	378	281	281
query78	11832	11766	11261	11261
query79	1432	1238	735	735
query80	1166	557	482	482
query81	515	345	288	288
query82	569	154	132	132
query83	393	332	309	309
query84	334	155	126	126
query85	1002	600	547	547
query86	408	299	258	258
query87	1810	1855	1743	1743
query88	3675	2783	2784	2783
query89	449	368	333	333
query90	1777	202	198	198
query91	205	199	168	168
query92	60	61	56	56
query93	1607	1533	976	976
query94	638	353	318	318
query95	791	581	480	480
query96	1025	819	361	361
query97	2650	2646	2497	2497
query98	210	213	205	205
query99	1098	1112	978	978
Total cold run time: 262694 ms
Total hot run time: 177743 ms

[hello-stephen]

ClickBench: Total hot run time: 24.95 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 25e389847049906c14d3d6bbf80f5c7ed876b390, data reload: false

query1	0.01	0.01	0.00
query2	0.10	0.05	0.06
query3	0.26	0.14	0.14
query4	1.60	0.14	0.14
query5	0.24	0.22	0.23
query6	1.23	1.06	1.02
query7	0.04	0.01	0.01
query8	0.05	0.04	0.03
query9	0.39	0.32	0.31
query10	0.55	0.55	0.54
query11	0.18	0.14	0.15
query12	0.19	0.14	0.14
query13	0.48	0.47	0.48
query14	1.00	1.00	1.01
query15	0.61	0.58	0.60
query16	0.31	0.35	0.31
query17	1.05	1.09	1.15
query18	0.23	0.21	0.21
query19	1.97	1.88	1.90
query20	0.01	0.01	0.01
query21	15.45	0.23	0.13
query22	4.68	0.05	0.06
query23	16.12	0.32	0.13
query24	2.97	0.42	0.32
query25	0.11	0.05	0.07
query26	0.76	0.20	0.14
query27	0.05	0.03	0.03
query28	3.51	0.94	0.55
query29	12.49	4.12	3.30
query30	0.29	0.15	0.15
query31	2.79	0.60	0.30
query32	3.23	0.59	0.49
query33	3.18	3.20	3.25
query34	15.56	4.22	3.49
query35	3.55	3.49	3.53
query36	0.56	0.44	0.45
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.19	0.18	0.16
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.32 s
Total hot run time: 24.95 s

[hello-stephen]

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

[nsivarajan]

run feut

[hello-stephen]

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report