Skip to content

cli: give infoctl query a QueryContext, a single-pass load and a memory limit #95

Open

Description

@codecapo

Before infoctl query (#15) is called stable, engine::query should take a QueryContext, read every table from heads loaded in one pass, and run under a memory limit and a row cap, and the command should get a short RFC. The command is still a stub at a0b5e6c (crates/cli/src/main.rs:18-20) and crates/engine/src/lib.rs:1-2 holds only a doc comment, so this work lands on the code PR #46 adds. There, engine::query(warehouse, sql) takes no caller identity. It registers one IcebergStaticTableProvider per table, each pinned to the head that its own Warehouse::load call returned inside the registration loop (load is private at crates/storage/src/lib.rs:296-301, and the PR makes it public). It builds its session with SessionContext::new(), whose DataFusion memory pool is unbounded, and it collects the whole result into memory that the pool does not count. A job commits its tables one at a time in a fixed order (docs/adr/0006-ordered-job-commits.md:41-46), because iceberg-rust has no multi-table commit (docs/adr/0006-ordered-job-commits.md:13-17). So a job that commits while those loads run can leave a join reading one table from before its commit and another from after it. Loading every head before registering anything narrows that window but cannot close it, so the gap that remains is documented, as ADR 0006 already does for quarantine rows (docs/adr/0006-ordered-job-commits.md:67-70).

The context holds one local principal and an empty on_behalf_of slot. Running queries with the user's own permissions (docs/architecture.md:115-117), per-table authorization and the layer read rule for models (docs/rfcs/0001-project-files.md:135-136) can then come later without another signature change. The limits guard usability on a laptop and are not a security boundary, since denial of service through very expensive queries is out of scope (SECURITY.md:28-30). This can go into PR #46 if it is still open, or straight after it. The deny-list tests and the catalog.db check are #93. The execution RFC in #74 settles the QueryContext shape. Because the struct added here is #[non_exhaustive] and built by a constructor, that RFC can add fields later without another signature change.

Crates: crates/engine and crates/cli (the query code and tests PR #46 adds), crates/storage only if the load pass gets its own Warehouse method, and a new RFC under docs/rfcs/

Done when

  • engine::query takes a &QueryContext, a #[non_exhaustive] struct built by a constructor such as QueryContext::local(). It holds one local principal and an on_behalf_of slot left as None for now. infoctl query passes it, and a test calls engine::query with it.
  • engine::query loads the head of every table in one pass before it builds or registers any table provider, then registers each table from the Table that pass returned.
  • The engine::query doc comment drops the claim that each table is read at the snapshot it had when the query started. It says instead that each table is read at the head the load pass returned, and that the pinned iceberg-rust gives no atomicity across tables, so a query can still see one of a job's tables before the job's commit and another after it.
  • Test: after the first of two tables is registered, a hook commits new rows to the second table, and a join of the two returns only rows from the heads the load pass returned. Against the per-table loads in PR feat(cli): infoctl query runs read-only SQL against the warehouse #46, the same test sees the new rows.
  • engine::query builds its session on a bounded DataFusion memory pool (RuntimeEnvBuilder::with_memory_limit) instead of the unbounded default. The limit has a default and an infoctl query option to change it. A test shows that with a very small limit an order by over a test table fails with an error that names the limit and the option, and that infoctl query then exits 1.
  • infoctl query stops reading once a row cap is reached instead of collecting the whole result, for example by asking DataFusion for one row more than the cap. It prints at most the cap and notes on stderr that the output was cut, so that CSV and JSON on stdout stay valid. A test with more rows than a small cap checks every --format.
  • A short infoctl query RFC, copied from docs/rfcs/0000-template.md (CONTRIBUTING.md:80-82), settles the command's options, output formats and exit codes and the default memory limit and row cap, and links to the execution RFC in dist: write an RFC for the Job, JobOutcome and runner contracts (decision D3) #74 for the QueryContext shape. Two maintainers approve it after at least one week open for comment (GOVERNANCE.md:28-29), before the command is called stable.
  • If the single-pass load changes a public interface, for example with a new Warehouse method that loads several heads, the RFC covers that change too, and the change merges only after the RFC is approved.

Points: 5 (an estimate: C25, C31 and S5-10 carry no points in the design review of 2026-11-01)

Depends on: #15 (the infoctl query code this builds on)
Blocks: nothing

Source: design review of 2026-11-01, findings C25 and C31, section 5.3 (S5-10) and section 8.5.

Activity

  1. added this to the 0.1.0 milestone on Nov 6, 2026
  2. codecapo commented on Nov 10, 2026

    @codecapo
    OwnerAuthor

    RFC 0002 (#102) fixes the QueryContext shape: a #[non_exhaustive] struct in crates/model with local: Principal and on_behalf_of: Option<Principal>, built by QueryContext::local(). Principal is a #[non_exhaustive] enum with one variant, Local, the OS user running the process. The RFC also maps DataFusion errors by find_root to Error::Parse, Plan, Data or Execution, which engine::query can reuse. The short infoctl query RFC this ticket asks for links to RFC 0002 for the context shape.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions