Skip to content

[default values] Add an overridable hook to transform table metadata as it is loaded#641

Merged
cbb330 merged 1 commit into
linkedin:mainfrom
cbb330:chbush/column-default-read-overlay
Jun 26, 2026
Merged

[default values] Add an overridable hook to transform table metadata as it is loaded#641
cbb330 merged 1 commit into
linkedin:mainfrom
cbb330:chbush/column-default-read-overlay

Conversation

@cbb330

@cbb330 cbb330 commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

This adds a small extension point to the OpenHouse client so a subclass can adjust a table's metadata as it is read in, without changing how tables are refreshed or cached. By default nothing changes — the metadata is read exactly as before. This is the open-source, source-agnostic half of a larger effort; the part that derives and attaches LinkedIn-specific column defaults lives in a separate internal client (which overrides this hook) and is intentionally not in this repository.

Terms: metadata.json is the file describing a table (its columns, schema history, snapshots). TableMetadata is the in-memory object parsed from it. doRefresh() is the OpenHouse client method that loads that metadata from storage. current() is the standard call a reader makes to get a table's current metadata.

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

Internal API Changes + Refactoring — an overridable metadata-load step.

  • OpenHouseTableOperations now reads metadata through one overridable method. A new protected TableMetadata loadMetadata(String metadataLocation) performs the read, and doRefresh() calls it instead of parsing inline. The default implementation reads the metadata as-is with the stock parser (TableMetadataParser.read).
  • Behavior is unchanged. doRefresh() previously called the one-argument refreshFromMetadataLocation(location). It now calls the four-argument form with (shouldRetry=null, numRetries=20) — which is exactly what the one-argument form does internally — with the final parse step routed through loadMetadata so it can be overridden.
  • Why route it through a subclass method. Because the override runs inside the loader, whatever it returns becomes the metadata the table stores and reports, so current() stays stock and there is no second, cached copy that could drift out of sync. An override owns the entire read-in, so it can also rewrite the stored bytes before parsing — useful for future format-compatibility shims.

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. The change is behavior-preserving: the default loadMetadata performs the same stock parse doRefresh() already did, and that path is exercised by the existing OpenHouseTableOperations / doRefresh tests. The override path is tested where it is implemented, in the internal client.
  • Some other form of testing like staging or soak time in production. Please explain.

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

This is one half of a larger effort to read column defaults that are declared in a table property rather than in the Iceberg metadata file. This PR contains only the general, source-agnostic extension point. The companion change — which overrides loadMetadata to derive those defaults and attach them to the schema — lives in a separate internal client, along with its dependencies, and is intentionally kept out of this repository. No new dependencies; no change to the commit path, caching, or metrics.

@cbb330 cbb330 force-pushed the chbush/column-default-read-overlay branch from 7702e24 to d939687 Compare June 26, 2026 18:53
@cbb330 cbb330 changed the title Apply declared column defaults when reading OpenHouse tables Add an overridable hook to transform table metadata as it is loaded Jun 26, 2026
Route the metadata read-in performed during doRefresh through a new protected loadMetadata(String) method. The default reads the metadata as-is with the stock parser, so behavior is unchanged: the 4-arg refreshFromMetadataLocation call with (shouldRetry=null, numRetries=20) is exactly the stock 1-arg form.

Subclasses can override loadMetadata to transform table metadata as it is loaded -- for example to attach column defaults declared in a table property -- without changing how the table is refreshed or cached. Because the transform runs inside the loader, whatever it returns becomes the metadata the base class stores and current() stays stock, so there is no separate cache that could drift.
@cbb330 cbb330 force-pushed the chbush/column-default-read-overlay branch from d939687 to 68922be Compare June 26, 2026 19:07
@cbb330 cbb330 marked this pull request as ready for review June 26, 2026 19:52
@cbb330 cbb330 merged commit 1c8dce1 into linkedin:main Jun 26, 2026
1 check passed
@cbb330 cbb330 changed the title Add an overridable hook to transform table metadata as it is loaded [default values] Add an overridable hook to transform table metadata as it is loaded Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants