Skip to content

Catalog Migrator: Add Iceberg view migration support - #265

Open
ajantha-bhat wants to merge 1 commit into
apache:mainfrom
ajantha-bhat:views
Open

Catalog Migrator: Add Iceberg view migration support#265
ajantha-bhat wants to merge 1 commit into
apache:mainfrom
ajantha-bhat:views

Conversation

@ajantha-bhat

Copy link
Copy Markdown
Member

Summary

Adds Iceberg view migration support to the Iceberg Catalog Migrator.

The migrator now supports migrating both Iceberg tables and Iceberg views from a source catalog to a target catalog without copying data. View migration uses the new Iceberg ViewCatalog.registerView support added in Iceberg 1.11.0, which registers a view in the target catalog from the source view metadata location.

By default, the migrate command now migrates both tables and views when both source and target catalogs support Iceberg views. Existing table-only selector behavior is preserved for users who explicitly provide table selectors.

New Options

Adds view selector options to the migrate command:

Option Description
--view-identifiers Comma-separated list of view identifiers to migrate
--view-identifiers-from-file File containing one view identifier per line
--view-identifiers-regex Regex used to select matching view identifiers

Selector behavior:

Table selector View selector What happens
Not specified Not specified Migrates all tables and all views
Specified Not specified Migrates selected tables only, no views
Not specified Specified Migrates selected views only, no tables
Specified Specified Migrates selected tables and selected views

Why Views Are Supported Only With migrate

View support is intentionally added only to the migrate command, not the register command.

For tables, register can leave the table entry in both source and target catalogs. That is already risky and documented, but table cleanup can use dropTable(..., purge=false) when migrating so only the catalog entry is removed from the source.

Iceberg views do not have an equivalent purge-false delete operation. Dropping a view is a lightweight catalog operation, so view migration must delete the successfully migrated view from the source catalog after registering it in the target catalog. Because of that, view migration requires migrate semantics and is rejected for register.

Notes

  • View migration runs only when both source and target catalogs implement Iceberg view support.
  • Table migration and view migration are tracked independently in the result summary.
  • Failures for tables or views are reported separately, and failed identifiers can be retried using the existing table selector options or the new view selector options.
  • Documentation and examples are updated to show table and view migration usage.

@ajantha-bhat

Copy link
Copy Markdown
Member Author

@dimas-b: Please take a look when you have time. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants