[default values] Tables: apply server-supplied per-column defaults at read time by cbb330 · Pull Request #645 · linkedin/openhouse

cbb330 · 2026-06-28T23:27:37Z

Summary

Adds the open-source read-bridge feature on top of the per-table config channel (#644): the OH server stamps per-column initial-defaults onto GetTableResponseBody.config, and the Java client overlays them at metadata-load time so a column added after data exists reads its declared default instead of NULL (a v2→v3 read-time bridge).

The whole feature is open source; it exposes exactly one deployment seam:

ColumnDefaultsSource (interface) — the single open-source/closed-source line: field-id -> Iceberg single-value JSON. The open-source default is a no-op lambda in ApiConfig; a deployment overrides this bean (li-openhouse derives values from avro.schema.literal).
ReadBridgeConfigResolver (server) — stamps each default as a flat, namespaced config entry: openhouse.read-bridge.column-default.<fieldId> = <single-value-json>. No envelope/POJO — the REST config string map carries the structure directly.
ReadBridge (client) — decodes those entries and applies the read-time overlay in OpenHouseTableOperations.loadMetadata (the withInitialDefault/withSchemaOverlay transform is a marked TODO). loadMetadata stays one delegating call.

Scope (engines)

Spark 3.1 (iceberg-1.2): this PR. The client overlay is in the iceberg-1.2 runtime only.
Spark 3.5 / iceberg-1.5: not supported — intentionally dropped. We don't need read-bridge there.
Flink: separate follow-up PR (if/when a Flink read path needs the overlay).

Stack

#644 (config channel) → #645 (read-bridge mechanism) → li-openhouse #2166 (li column-default source from avro.schema.literal)

Stacked on #644 — review/merge after #644.

Next steps (this PR)

Implement the overlay in ReadBridge.apply — withInitialDefault / withSchemaOverlay (currently a TODO returning raw); pure + fail-closed.
Rebase once [default values] Tables: add a server-stamped per-table config map to GetTableResponseBody and capture it client-side #644 merges so the diff narrows to the read-bridge commit.

Testing Done

:services:tables + iceberg-1.2 runtime compile (JDK 17).
ReadBridgeConfigResolverTest (server): each default round-trips as a flat openhouse.read-bridge.column-default.<fieldId> entry; empty when the source supplies nothing.
ReadBridgeTest (client): decodes the flat entries by field-id and skips bad field-ids / unparseable values. OpenHouseTableOperationsTest config capture/deserialization pass. Repo-wide spotlessCheck green.

shanthoosh · 2026-07-01T02:43:54Z

+    }
+    // TODO(read-bridge): overlay columnDefaults onto raw.schemas() via withSchemaOverlay; future V3
+    // features bridged from config are applied here too.
+    return raw;


This method is currently a no-op. The apply() method currently always returns raw unchanged, and I couldn’t find a call site where this method is invoked during table metadata refresh/load. The server stamps the read-bridge config into LoadTableResponse, but the client never consumes it, so the feature isn’t active end-to-end. Do we plan to extend this further in the future PRs as a follow-up?

Yes, by design this #645 is the O.S. substrate and the overlay transform marked TODO, and defaults only arrive once a ColumnDefaultsSource is supplied in closed/source since it relies on linkedin schema object "avro.schema.literal"

shanthoosh · 2026-07-01T02:46:02Z

+        int fieldId = Integer.parseInt(entry.getKey().substring(COLUMN_DEFAULT_PREFIX.length()));
+        byFieldId.put(fieldId, MAPPER.readTree(entry.getValue()));
+      } catch (RuntimeException | JsonProcessingException e) {
+        log.warn(


Minor: It would be useful to log the COLUMN_DEFAULT_PREFIX to aid debugging

shanthoosh · 2026-07-01T02:50:43Z

+      try {
+        int fieldId = Integer.parseInt(entry.getKey().substring(COLUMN_DEFAULT_PREFIX.length()));
+        byFieldId.put(fieldId, MAPPER.readTree(entry.getValue()));
+      } catch (RuntimeException | JsonProcessingException e) {


Is it safe to silently not consider the (key, value) pairs that fails with exception and chose the other values. Are we expecting a fixed set of configuration key as json in this case? If so, would be reasonable to set default if the values for config-key are malformed?

shanthoosh

Looks good overall. Have few minor clarification questions/comments.

… GetTableResponseBody and capture it client-side (#644) ## Summary Adds a generic, server-stamped, per-table client **`config`** map to `GetTableResponseBody` and captures it in `OpenHouseTableOperations` so subclasses can read it. It follows the **Iceberg REST `LoadTableResponse.config` convention** — a `Map<String,String>` of client-side behavior overrides the server controls at runtime, without re-rolling the slow-to-upgrade Java client fleet. **No behavior change**: the map is null/empty until a server stamps it, and the base client has nothing to act on. ## Changes - [x] Client-facing API Changes - [x] Internal API Changes - [ ] Bug Fixes - [x] New Features - [x] Tests **Client-facing API Changes** - READ_ONLY, nullable `config` field (`Map<String,String>`) on `GetTableResponseBody`, modeled on the Iceberg REST load-table `config`. Namespaced keys (e.g. `openhouse.read-bridge`); clients ignore keys they do not understand. - Why a response field: `doRefresh` already fetches a live `GetTableResponseBody` on every table load (and on commit responses) but discarded all but `getTableLocation()` — it is the natural, already-present, server-controlled, zero-staleness delivery channel. The generated client sets `FAIL_ON_UNKNOWN_PROPERTIES = false`, so a new response field cannot break older clients (and unknown config keys are simply carried). Same additive, nullable, READ_ONLY pattern as `sortOrder` etc. **Internal API Changes** - `OpenHouseTableOperations.doRefresh` keeps the full response and stashes `config` in an `AtomicReference`, exposed to subclasses via the new `protected currentConfig()`. READ_ONLY + side-channel: never sent back on writes. - `TablesMapper` (the table DTO serializer into gettableresponsebody) ignores `config` (`@Mapping(target = "config", ignore = true)`) — it is stamped separately, not sourced from `TableDto`. **New Features** - A flat string map keyed by namespaced keys means new features become new `config` entries rather than an API/schema change or client regen. ## Testing Done - [x] Added new tests. Compile/codegen verified (`:services:tables`, `:client:tableclient`, `:integrations:java:iceberg-1.2:openhouse-java-runtime`); the client regenerates with `getConfig()` returning `Map<String,String>`. `OpenHouseTableOperationsTest` covers: `currentConfig()` null before refresh; `doRefresh` captures/clears `config`; the REST-style string map deserializes from a response and tolerates unknown fields. ## Stack **#644 (channel)** → #645 (read-bridge mechanism) → [li-openhouse #2166](https://github.com/linkedin-multiproduct/li-openhouse/pull/2166) (li column-default source from avro.schema.literal) This is the substrate PR — intentionally behaviorless on its own. ## Next steps (this PR) - [ ] Review + merge first — it is the base of the stack; #645 and #2166 depend on it. This PR delivers only the channel; it stays inert until two further pieces exist, both **required**, and both delivered by the next PR in the stack: - **A server stamp** to populate `config` (it is null until something stamps it) — required, delivered by #645 via `ReadBridgeConfigResolver` + the `ColumnDefaultsSource` seam. - **A client consumer** that reads `currentConfig()` and acts on it — required, delivered by #645's read-bridge metadata overlay, driven by the column defaults that #2166 supplies. --------- Co-authored-by: Cursor <cursoragent@cursor.com>

The base branch was changed.

Adds the open-source read-bridge feature on top of the per-table `config` channel (linkedin#644): - ColumnDefaultsSource: the single open-source/closed-source seam (field-id -> Iceberg single-value JSON). Open-source default is a no-op lambda in ApiConfig; a deployment overrides this bean (e.g. li-openhouse, from avro.schema.literal). - ReadBridgeConfigResolver: server-side encoder that stamps each default as a flat namespaced config entry (openhouse.read-bridge.column-default.<fieldId> = single-value JSON) — no envelope/POJO; the config map carries the structure. - ReadBridge (client): decodes those entries and applies the overlay at metadata-load time (the column-default transform is a marked TODO, and the place further V3 features get backported). Keeps loadMetadata to one call. Behaviorless until a ColumnDefaultsSource is supplied; fail-closed throughout. Co-authored-by: Cursor <cursoragent@cursor.com>

cbb330 mentioned this pull request Jun 28, 2026

[default values] Tables: add a server-stamped per-table config map to GetTableResponseBody and capture it client-side #644

Merged

7 tasks

cbb330 changed the title ~~Tables: read-bridge column-default bridge — ColumnDefaultsSource seam + client overlay~~ [default values] Tables: read-bridge column-default bridge — ColumnDefaultsSource seam + client overlay Jun 29, 2026

cbb330 force-pushed the chbush/feature-flags-resolver branch from c6641c7 to bc2d69c Compare June 29, 2026 02:44

cbb330 changed the title ~~[default values] Tables: read-bridge column-default bridge — ColumnDefaultsSource seam + client overlay~~ [default values] Tables: read-bridge — ColumnDefaultsSource seam + read-time column-default overlay Jun 29, 2026

cbb330 changed the title ~~[default values] Tables: read-bridge — ColumnDefaultsSource seam + read-time column-default overlay~~ [default values] Tables: apply server-supplied per-column defaults at read time Jun 29, 2026

cbb330 changed the base branch from main to chbush/runtime-policy-response June 29, 2026 03:45

cbb330 force-pushed the chbush/feature-flags-resolver branch 4 times, most recently from 9b5e906 to 14c82c3 Compare June 29, 2026 05:27

cbb330 force-pushed the chbush/runtime-policy-response branch from 7c276f0 to fdfaf48 Compare June 29, 2026 22:26

cbb330 force-pushed the chbush/feature-flags-resolver branch 4 times, most recently from d4ac8fd to 7df59f3 Compare June 30, 2026 00:18

cbb330 marked this pull request as ready for review June 30, 2026 00:23

shanthoosh reviewed Jul 1, 2026

View reviewed changes

shanthoosh previously approved these changes Jul 1, 2026

View reviewed changes

cbb330 force-pushed the chbush/feature-flags-resolver branch from 7df59f3 to 747c7ef Compare July 1, 2026 21:29

cbb330 force-pushed the chbush/feature-flags-resolver branch from 747c7ef to 112cdab Compare July 4, 2026 15:36

cbb330 changed the base branch from chbush/runtime-policy-response to main July 4, 2026 15:36

cbb330 force-pushed the chbush/feature-flags-resolver branch from 112cdab to 2f2fccb Compare July 4, 2026 15:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[default values] Tables: apply server-supplied per-column defaults at read time#645

[default values] Tables: apply server-supplied per-column defaults at read time#645
cbb330 wants to merge 1 commit into
linkedin:mainfrom
cbb330:chbush/feature-flags-resolver

cbb330 commented Jun 28, 2026 •

edited

Loading

Uh oh!

shanthoosh Jul 1, 2026 •

edited

Loading

Uh oh!

cbb330 Jul 4, 2026

Uh oh!

shanthoosh Jul 1, 2026

Uh oh!

shanthoosh Jul 1, 2026

Uh oh!

shanthoosh left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

cbb330 commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope (engines)

Stack

Next steps (this PR)

Testing Done

Uh oh!

shanthoosh Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cbb330 Jul 4, 2026

Choose a reason for hiding this comment

Uh oh!

shanthoosh Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

shanthoosh Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

shanthoosh left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cbb330 commented Jun 28, 2026 •

edited

Loading

shanthoosh Jul 1, 2026 •

edited

Loading