[default values] Add an overridable hook to transform table metadata as it is loaded#641
Merged
Merged
Conversation
7702e24 to
d939687
Compare
Route the metadata read-in performed during doRefresh through a new protected loadMetadata(String) method. The default reads the metadata as-is with the stock parser, so behavior is unchanged: the 4-arg refreshFromMetadataLocation call with (shouldRetry=null, numRetries=20) is exactly the stock 1-arg form. Subclasses can override loadMetadata to transform table metadata as it is loaded -- for example to attach column defaults declared in a table property -- without changing how the table is refreshed or cached. Because the transform runs inside the loader, whatever it returns becomes the metadata the base class stores and current() stays stock, so there is no separate cache that could drift.
d939687 to
68922be
Compare
shanthoosh
approved these changes
Jun 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds a small extension point to the OpenHouse client so a subclass can adjust a table's metadata as it is read in, without changing how tables are refreshed or cached. By default nothing changes — the metadata is read exactly as before. This is the open-source, source-agnostic half of a larger effort; the part that derives and attaches LinkedIn-specific column defaults lives in a separate internal client (which overrides this hook) and is intentionally not in this repository.
Terms:
metadata.jsonis the file describing a table (its columns, schema history, snapshots).TableMetadatais the in-memory object parsed from it.doRefresh()is the OpenHouse client method that loads that metadata from storage.current()is the standard call a reader makes to get a table's current metadata.Changes
Internal API Changes + Refactoring — an overridable metadata-load step.
OpenHouseTableOperationsnow reads metadata through one overridable method. A newprotected TableMetadata loadMetadata(String metadataLocation)performs the read, anddoRefresh()calls it instead of parsing inline. The default implementation reads the metadata as-is with the stock parser (TableMetadataParser.read).doRefresh()previously called the one-argumentrefreshFromMetadataLocation(location). It now calls the four-argument form with(shouldRetry=null, numRetries=20)— which is exactly what the one-argument form does internally — with the final parse step routed throughloadMetadataso it can be overridden.current()stays stock and there is no second, cached copy that could drift out of sync. An override owns the entire read-in, so it can also rewrite the stored bytes before parsing — useful for future format-compatibility shims.Testing Done
loadMetadataperforms the same stock parsedoRefresh()already did, and that path is exercised by the existingOpenHouseTableOperations/doRefreshtests. The override path is tested where it is implemented, in the internal client.Additional Information
This is one half of a larger effort to read column defaults that are declared in a table property rather than in the Iceberg metadata file. This PR contains only the general, source-agnostic extension point. The companion change — which overrides
loadMetadatato derive those defaults and attach them to the schema — lives in a separate internal client, along with its dependencies, and is intentionally kept out of this repository. No new dependencies; no change to the commit path, caching, or metrics.