Repository navigation
Description
Before infoctl query (#15) is called stable, engine::query should take a QueryContext, read every table from heads loaded in one pass, and run under a memory limit and a row cap, and the command should get a short RFC. The command is still a stub at a0b5e6c (crates/cli/src/main.rs:18-20) and crates/engine/src/lib.rs:1-2 holds only a doc comment, so this work lands on the code PR #46 adds. There, engine::query(warehouse, sql) takes no caller identity. It registers one IcebergStaticTableProvider per table, each pinned to the head that its own Warehouse::load call returned inside the registration loop (load is private at crates/storage/src/lib.rs:296-301, and the PR makes it public). It builds its session with SessionContext::new(), whose DataFusion memory pool is unbounded, and it collects the whole result into memory that the pool does not count. A job commits its tables one at a time in a fixed order (docs/adr/0006-ordered-job-commits.md:41-46), because iceberg-rust has no multi-table commit (docs/adr/0006-ordered-job-commits.md:13-17). So a job that commits while those loads run can leave a join reading one table from before its commit and another from after it. Loading every head before registering anything narrows that window but cannot close it, so the gap that remains is documented, as ADR 0006 already does for quarantine rows (docs/adr/0006-ordered-job-commits.md:67-70).
The context holds one local principal and an empty on_behalf_of slot. Running queries with the user's own permissions (docs/architecture.md:115-117), per-table authorization and the layer read rule for models (docs/rfcs/0001-project-files.md:135-136) can then come later without another signature change. The limits guard usability on a laptop and are not a security boundary, since denial of service through very expensive queries is out of scope (SECURITY.md:28-30). This can go into PR #46 if it is still open, or straight after it. The deny-list tests and the catalog.db check are #93. The execution RFC in #74 settles the QueryContext shape. Because the struct added here is #[non_exhaustive] and built by a constructor, that RFC can add fields later without another signature change.
Crates: crates/engine and crates/cli (the query code and tests PR #46 adds), crates/storage only if the load pass gets its own Warehouse method, and a new RFC under docs/rfcs/
Done when
-
engine::querytakes a&QueryContext, a#[non_exhaustive]struct built by a constructor such asQueryContext::local(). It holds onelocalprincipal and anon_behalf_ofslot left asNonefor now.infoctl querypasses it, and a test callsengine::querywith it. -
engine::queryloads the head of every table in one pass before it builds or registers any table provider, then registers each table from theTablethat pass returned. - The
engine::querydoc comment drops the claim that each table is read at the snapshot it had when the query started. It says instead that each table is read at the head the load pass returned, and that the pinned iceberg-rust gives no atomicity across tables, so a query can still see one of a job's tables before the job's commit and another after it. - Test: after the first of two tables is registered, a hook commits new rows to the second table, and a join of the two returns only rows from the heads the load pass returned. Against the per-table loads in PR feat(cli): infoctl query runs read-only SQL against the warehouse #46, the same test sees the new rows.
-
engine::querybuilds its session on a bounded DataFusion memory pool (RuntimeEnvBuilder::with_memory_limit) instead of the unbounded default. The limit has a default and aninfoctl queryoption to change it. A test shows that with a very small limit anorder byover a test table fails with an error that names the limit and the option, and thatinfoctl querythen exits 1. -
infoctl querystops reading once a row cap is reached instead of collecting the whole result, for example by asking DataFusion for one row more than the cap. It prints at most the cap and notes on stderr that the output was cut, so that CSV and JSON on stdout stay valid. A test with more rows than a small cap checks every--format. - A short
infoctl queryRFC, copied fromdocs/rfcs/0000-template.md(CONTRIBUTING.md:80-82), settles the command's options, output formats and exit codes and the default memory limit and row cap, and links to the execution RFC in dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 for theQueryContextshape. Two maintainers approve it after at least one week open for comment (GOVERNANCE.md:28-29), before the command is called stable. - If the single-pass load changes a public interface, for example with a new
Warehousemethod that loads several heads, the RFC covers that change too, and the change merges only after the RFC is approved.
Points: 5 (an estimate: C25, C31 and S5-10 carry no points in the design review of 2026-11-01)
Depends on: #15 (the infoctl query code this builds on)
Blocks: nothing
Source: design review of 2026-11-01, findings C25 and C31, section 5.3 (S5-10) and section 8.5.
Activity
codecapo commentedon Nov 10, 2026
RFC 0002 (#102) fixes the
QueryContextshape: a#[non_exhaustive]struct incrates/modelwithlocal: Principalandon_behalf_of: Option<Principal>, built byQueryContext::local().Principalis a#[non_exhaustive]enum with one variant,Local, the OS user running the process. The RFC also maps DataFusion errors byfind_roottoError::Parse,Plan,DataorExecution, whichengine::querycan reuse. The shortinfoctl queryRFC this ticket asks for links to RFC 0002 for the context shape.docs: add RFC 0002 for the job and runner contracts (#102)
feat(cli): infoctl query runs read-only SQL against the warehouse (#46)