feat(auth): authorize query-auth reads and carry the grant on the split

[plusplusjiajia]

Purpose

A query-auth.enabled table makes the server return a per-user row filter and column masking that the client is expected to apply. This client cannot apply them yet, so it refuses to read such a table at all — even for a user the server reports as unrestricted. This slice fetches the authorization at scan-plan time and carries it to the read, so that user can read. A user with rules gets the same refusal as before.

Brief change log

TableScan::plan authorizes once and stamps the result on every split, as Java wraps each split in a QueryAuthSplit. TableRead::to_arrow then decides from the splits: each must carry a grant, from this handle, and unrestricted.

Whether a table is query-auth comes from the server, not the loaded handle — the option can be set after a load, and a cached false would skip authorization entirely. Fourteen sites ask, including the searches, index builds and rewrites that read files without planning. Sync read boundaries cannot ask, so they read a marker the splits carry; it survives serialization, so a round-tripped split that lost its grant fails closed.

Five refusals are deliberate: a restricted grant, at planning, since a plan carries row counts and min/max that answer COUNT/MIN/MAX without reading a row; a time-travelled, branch or decorated (db.t$branch_x, db.t$files) handle, which reads files the server did not rule on; a field that is not an exact (id, name, type) match, catching an older shape with an extra nested child; a data file whose statistics name a column the current schema lacks, since value_stats is public on every split; and a read naming a reserved system column, which the server's own check would reject.

The request names no columns, which the server expands to the real schema fields — Java sends the read type's names, so a user authorized for only a subset is still refused here. AuthTableQueryResponse rejects unknown fields: an absent one reads as "no rule", so protocol drift would look like an unrestricted grant.

Known limitation

Authorization is a plan-time decision and a plan is the capability recording it. A split kept from before the option was enabled, or built by hand, is read on the caller's word — as in Java, where unwrapQueryAuthSplit returns no result for a plain split. Callers must re-plan after an authorization change.

A read-boundary guard could not change this: Table::file_io() and DataSplit::data_file_path() are public, so the same caller can read the files directly without going through TableRead.

[JingsongLi]

[P1] Apply the live server check to direct search APIs too

This helper closes the stale-handle gap for TableScan, but the direct scored/search entry points still call only CoreOptions::ensure_read_authorized() on the schema cached when the handle was loaded. In particular, BatchVectorSearchBuilder::execute reads the snapshot/index manifest directly, and VectorSearchBuilder::execute_scored, FullTextSearchBuilder::execute_scored, and HybridSearchBuilder::execute_scored reach those direct paths without an authorized TableScan.

Therefore: load a REST table while query auth is false, enable restricted query auth on the server, then reuse the handle for one of these searches. The cached guard passes and row IDs/scores derived from protected data are returned without the auth exchange. Please route every out-of-band search entry through this async server-state check and reject when query auth is enabled (these paths cannot apply masking/filtering), with stale-handle regressions analogous to the new scan test.

[plusplusjiajia]

@JingsongLi Good catch — real hole. Fixed and widened past the searches: fourteen sites now ask the server via Table::ensure_read_authorized_live, and delegated searches ask once rather than once per route.

[JingsongLi]

[P1] Validate the full nested field shape, not only the top-level pair

Both this guard and the old-file check above compare only the top-level (id, name). That leaves a concrete disclosure path after nested schema evolution: suppose the authorized current schema contains profile ROW<public>, while a live older file has the same top-level field id/name but profile ROW<public, secret>. A caller can use the public ReadBuilder::with_read_type with that old Row type; this check passes, and data_file_reader::prune_data_type recursively selects the requested old child by id, so profile.secret is decoded even though it is absent from the schema/column set the server authorized. The planning check at lines 87-92 also passes the old file for the same reason.

Please validate canonical fields recursively (including Row children and nested Array/Map/Multiset element types, allowing only explicitly safe evolution), and make the old-file containment check recursive too. An end-to-end test with a dropped nested field and a crafted old read type should be rejected.

[plusplusjiajia]

@JingsongLi Good catch. Both checks now compare id, name and full data type; DataType derives structural equality, so nested shapes are covered recursively. End-to-end test as you asked: a Row column read with a read type that keeps the authorized (id, name) but carries an extra nested child — refused.

[JingsongLi]

[P1] Do not accept an unmarked split from a stale REST handle

When a REST handle was loaded while query auth was false, core_options.query_auth_enabled() stays false. If the server later enables restricted query auth, a caller that bypasses TableScan can still pass an old unmarked plan—or a split made through the public DataSplitBuilder—to this synchronous to_arrow boundary. required is false, so the method returns rows without any auth RPC or grant. This contradicts the PR’s stated stale-handle guarantee; the new test covers planning after the toggle, but not this public read path.

Please make REST reads require evidence that the split was planned after a live server-state check even when that check said query auth was disabled (for example, a catalog-session-bound checked-plan capability distinct from query_auth_required), or make materialization able to re-authorize. Unmarked splits should remain acceptable only where no REST authorization boundary exists. Add a regression for: load with false, retain/build an unmarked split, enable a restricted response, then call to_arrow and require refusal.

[plusplusjiajia]

@JingsongLi The gap is real, but I could not find a fix worth its cost, so I documented it here and in the PR description.

I implemented your suggestion first. It broke four legitimate flows on ordinary, non-query-auth REST tables — including DataFusion's register_cow_target_table and the documented C paimon_plan_from_split_bytes round trip — because it required enumerating every path that rebuilds a planned split, and I missed several. It also proved nothing: DataSplit is publicly serde-serializable, so {query_auth_checked: true, query_auth_required: false} skips both /auth and grant validation.

And closing it buys little: Table::file_io() and DataSplit::data_file_path() are public, so the same caller can read the bytes without touching TableRead. A real boundary means binding data tokens to an authorization epoch at the data plane — server-side work not available to us. Java stops here too.

So the contract is plan-time: callers must re-plan after an authorization change. A split from a query-auth table still fails closed if it lost its grant.

Do you have a better idea?

[JingsongLi]

[P1] Re-authorize before the writer performs lazy reads

This live check runs only when commit starts, but TableWrite has already read table state by then. TableWrite::new still checks only the schema cached on the handle; on the first normal PK write, create_kv_writer calls scan_partition_sequence_numbers, which reads the latest snapshot/manifests, and a dynamic-bucket write additionally runs DynamicBucketAssigner::ensure_index_entries_loaded plus HashIndexFile::read. Thus a handle loaded while query-auth.enabled=false can be reused after the server enables restricted query auth: write_arrow_batch and prepare_commit read protected metadata/index contents before this check eventually rejects the commit. The dynamic path can even emit a replacement hash-index file containing hashes restored from the protected index, so rejecting only at commit does not undo the disclosure. Please put a live authorization check before the first async writer read (or add an auth-aware async initialization) and cover the stale-handle write path.

[JingsongLi]

[P1] Do not trust a disabled answer from a replacement table

When the cached schema is false, this accepts the current name’s schema without checking that it still has rest_env’s UUID. A stale handle can therefore miss the very transition this helper is meant to catch: load table A with auth disabled, enable restricted query auth for A, then drop/re-create the name as table B with auth disabled. current_table() now reports B’s false, ensure_read_authorized_live succeeds, and direct vector/full-text/index paths read A’s still-reachable files through the stale handle without any auth exchange. The comment says the live answer only strengthens, but a false from a different UUID does not constrain A at all. Please validate the UUID before accepting a live false (schema freshness can remain a separate policy) and add a stale-false/recreated-table regression.

[JingsongLi]

[P1] Check the live branch schema, not the base-table name

For a table obtained through copy_with_branch, self.schema and the managers point at the branch, but rest_env.current_table() still queries the base identifier stored when the original handle was loaded. If the branch handle cached query-auth.enabled=false, the branch is later changed to restricted auth, and the base schema remains false, this method returns false; authorize_read(false) then exits before its branch_reference refusal and the scan reads the branch without an auth exchange. The current branch tests only start with a locally true option, so they do not cover this stale-false case. Please query branch_identifier(self.branch()) for live state (or conservatively refuse REST branch handles) and add a branch-specific toggle regression.