Skip to content

Reject non-IMPLICIT authentication for HADOOP, HIVE and BIGQUERY connections - #6145

Open
youngrae81 wants to merge 1 commit into
apache:mainfrom
youngrae81:fix/federation-auth-type-validation
Open

youngrae81 wants to merge 1 commit into
apache:mainfrom
youngrae81:fix/federation-auth-type-validation

Conversation

@youngrae81

Copy link
Copy Markdown
Contributor

The Hadoop, Hive and BigQuery Metastore federation factories only support IMPLICIT authentication and throw IllegalStateException for any other type when the catalog is first used.
Catalog creation only checked that the connection type and the authentication type were each in their allow-lists, so a HADOOP connection with, for example, BEARER authentication was created, its secret was stored, and every later request against the catalog failed with a 500.

This adds ImplicitOnlyConnectionValidator, a CatalogConfigValidator that rejects a non-IMPLICIT authentication type for these three connection types with a 400 (HADOOP connections only support IMPLICIT authentication, but BEARER was requested).
Like the existing BigLakeCatalogValidator, it is picked up by CatalogConfigValidators and runs on create and update, before any secret is written, in line with the dev@ thread "Applying CatalogConfigValidator to federated catalog and storage STS endpoint validation". A missing authenticationParameters is left to the existing request validation (#5909), and other connection types are not affected.

Since it also runs on update, a catalog that was already created with such a combination now gets a 400 on update as well (UpdateCatalogRequest cannot change the connection, so this only affects pre-existing catalogs). Such a catalog cannot serve any request today, so the changelog entry says to drop and recreate it.

Two things I considered and want to flag:

  • The IMPLICIT-only rule is a property of the factories in extensions/federation/*, and this validator restates it in runtime/service. Putting a CatalogConfigValidator next to each factory would keep the rule in one place, but the three extensions would each need a bean and the rule is the same for all of them. I went with one validator here, matching BigLakeCatalogValidator; happy to move it if you prefer the per-extension form.
  • Validators run before the ENABLE_CATALOG_FEDERATION check, so with federation disabled a HADOOP+BEARER request now reports the authentication problem rather than "feature not enabled". BigLakeCatalogValidator already behaves this way.

Not included: the BigQuery factory also checks warehouse and gcp.bigquery.project-id only at first use. That could go into the same validator, but I kept this PR to the authentication type; happy to add it if preferred.

Tests: TestServices now installs the validator like production does. ManagementServiceTest gets a parameterized test for HADOOP, HIVE and BIGQUERY with BEARER authentication; all three fail on main (the catalog is created) and pass with this change, which also checks that the catalog is not persisted. ImplicitOnlyConnectionValidatorTest covers IMPLICIT, BEARER, missing authentication, ICEBERG_REST and catalogs without a connection.
./gradlew format compileAll and :polaris-runtime-service:check pass.

Fixes #6138

Checklist

AI assistance was used to prepare this change; I reviewed and tested it end to end.

…ections

The Hadoop, Hive and BigQuery Metastore federation factories only support
IMPLICIT authentication and throw IllegalStateException for any other type
when the catalog is first used. Catalog creation only checked that the
connection type and the authentication type were each allowed, so such a
catalog was created, its secret was stored, and every later request against it
failed with a 500.

A new CatalogConfigValidator rejects these combinations with a 400. Like the
other validators it runs on create and update, before any secret is written.

Fixes apache#6138

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HADOOP, HIVE and BIGQUERY external catalogs accept a non-IMPLICIT authentication type at creation and then fail every request with 500

1 participant