Repository navigation
chore(deps): ignore RUSTSEC-2026-0459 for the bundled SQLite - #99
Merged
Merged
Conversation
The advisory entered the RustSec database on 2026-11-09, so every `cargo deny check` now fails. SQLite before 3.54 can return uninitialised page-cache memory when a database is opened next to a crafted -wal file whose page size differs. The bundled copy (SQLite 3.46.0) comes from libsqlite3-sys 0.30.1, which sqlx-sqlite 0.8.6 pins, which iceberg-catalog-sql requires. The fix is libsqlite3-sys 0.39, and no sqlx release accepts it yet: sqlx-sqlite 0.9.0 caps it below 0.38. So no dependency update can clear the advisory today. Only catalog.db in the warehouse directory is opened through this SQLite, and Infostash is the only writer there. The ignore carries a reason and a comment saying when to remove it, like the zstd ones. Signed-off-by: Mayur Kale <hello@mayurkale.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 58 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #99 +/- ##
=======================================
Coverage 99.14% 99.14%
=======================================
Files 8 8
Lines 1048 1048
=======================================
Hits 1039 1039
Misses 9 9 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
codecapo
added a commit
that referenced
this pull request
Nov 10, 2026
## What Two architecture decision records from the #72 spike on the replace commit route (decision D1), plus their rows in the ADR index. - **ADR 0007** chooses the hand-built route for 0.1.0: the `overwrite` snapshot is built from public iceberg-rust APIs and committed with our own one-row compare-and-swap on the SQL catalog's `iceberg_tables` row, with `TableRequirement::check` on the UUID and head first and zero rows updated mapped to `Error::Conflict` with no retry. It records the `cargo tree -d` result (one copy each of `iceberg` and `sqlx`), the state of upstream at the pin, how far the fork route got and why it is not taken, and the costs of the chosen route. - **ADR 0008** records the rule for incremental readers of a replaced table: when any input holds a non-append snapshot or an expired `from` snapshot, the scheduler pins every input of the model with `from = None`, reports the fallback and replaces the target as a new job; the refusal in `appends_between` stays as a guard that ends the job without a retry; the compile-time rule from #73 and #84 applies until the plan-time check from #89 lands. Closes #72 ## How The spike built a throwaway prototype in `crates/storage` on a local branch and committed the route with twelve tests: two replaces prepared on one head give one commit and one conflict with the loser's files referenced by no snapshot, and exactly one wins when they commit concurrently; a replace racing an append that lands first is a conflict; the table keeps its UUID, snapshot history and metadata log; a reader holding the old metadata still reads the old files; iceberg-rust's own `fast_append` builds on the hand-made snapshot; the incremental refusal fires on the real overwrite; a full refresh with no rows still commits; and a replace commits on a catalog whose `iceberg_tables` has no `iceberg_type` column, the older layout the catalog reads and appends to. Each head guard was then tried on its own against those tests: the swap alone still gives one winner, the requirement check alone leaves the window between check and swap open, so ADR 0007 asks #88 for a test per guard. The fork route was probed alongside with a one-line `[patch]` that makes `TableCommit::build` public; the same snapshot committed through `Catalog::update_table`, the catalog's own conflict refused a stale commit when the requirements were set, and without them a stale commit given a fresh sequence number landed over the moved head. Both ADRs record what those runs showed. ## Things to know - The commit statement is the SQL catalog's own less its record type filter: the table's key and the metadata location already pin the row, so the statement fits both catalog layouts. - The prototype is not in this pull request. #88 builds `Warehouse::replace_if_head` to the design ADR 0007 records. - The route keeps every untouched manifest, as #88's acceptance already asks, and never copies an earlier snapshot's `DELETED` entries into a manifest it writes. - A losing replace leaves manifests, a manifest list and a metadata file as well as data files; ADR 0007 says removal of unreferenced files must cover them, which widens #40's current scope. - `cargo deny check` is red on every branch until #99 lands (RUSTSEC-2026-0459, tracked in #98). Nothing here touches dependencies. - Only `docs/adr/` changes. No code, CI or configuration is touched. ## Tests - `typos docs/adr/` passes. - Every relative link in the two ADRs and the index resolves. - Both ADRs follow the format in `docs/adr/README.md`. --------- Signed-off-by: Mayur Kale <hello@mayurkale.com>
codecapo
added a commit
that referenced
this pull request
Nov 10, 2026
Closes #15 Closes #93 ## What `infoctl query "<sql>"` runs one read-only SQL statement against the warehouse and prints the result. ``` infoctl query [--warehouse <dir>] [--format table|csv|json] "<sql>" ``` - Tables are named `<layer>.<table>`, either by namespace (`gold.daily_revenue`) or by alias (`ready.daily_revenue`), as in ADR 0003. - Writes are rejected through DataFusion's `SQLOptions`: DML, DDL, `COPY TO`, `CREATE EXTERNAL TABLE` and `SET` all fail with "not supported", also under `EXPLAIN ANALYZE`, and the tests check that nothing was written. - `--format csv` prints a header. `--format json` prints one JSON object per row (line-delimited), with explicit nulls. An empty result still shows its columns in table and CSV output. - An unknown table gives `not found: table ready.missing; tables: gold.daily_revenue`, or says the warehouse has no tables. - Exit codes: 0 on success, 1 for a query error, 2 for a usage error. ## How - **engine:** `infostash_engine::query(&Warehouse, sql) -> QueryResult` registers each table as an `IcebergStaticTableProvider`. Every table is read at the snapshot it had when the query started. Each layer's schema is registered under both its namespace and its alias. - **storage:** new public `Warehouse::tables()` (every table in the three layer namespaces, sorted) and `Warehouse::load()` (previously private). - **model:** `Layer::ALL` and `Layer::alias()` sit next to `Layer::namespace`, so the ADR 0003 mapping stays in one place. - **cli:** the logic lives in `src/lib.rs` (`run(args, out, err) -> i32`), so it can be tested in-process. `main.rs` only passes the arguments through. - **deps:** `arrow-json = "59.2"` added to the workspace. It was already in the lock file through DataFusion. ## Things to know - **`--warehouse` is a stopgap.** It defaults to `.infostash/warehouse`, the value in the RFC 0001 draft. Once project files land (#5, #7), the warehouse should come from the project file. The flag is not in #15. - **A warehouse must already exist.** A missing directory, or a directory without `catalog.db`, is an error and nothing is created. `Warehouse::open_local` opens SQLite with `mode=rwc`, so the command checks for the catalog file before opening (#93). - **Scans run on one core.** That is the upstream provider's single partition, which #38 fixes. This PR is only about correctness. - **Follow-ups.** SQL that starts with a `--` comment is refused as an unknown option until #96 adds an end-of-options marker. `show tables` needs `information_schema`, which #97 turns on. A `QueryContext`, a single-pass load and a memory limit are #95. ## Tests - `crates/engine/tests/query.rs` (12 tests): - namespace and alias resolve to the same table - aggregates - an empty result keeps its schema - a table with no snapshot reads as empty - writes are rejected (`COPY TO`, `INSERT`, `CREATE TABLE AS`, `DROP`, `SET`) - `CREATE EXTERNAL TABLE`, `EXPLAIN ANALYZE INSERT` and a file path in `FROM` are rejected - the unknown-table message and the empty-warehouse message - parse errors and execution errors - `crates/cli/tests/query.rs` (8 tests) runs the real binary against a small committed table: table, CSV and JSON output, empty results, a rejected write, an unknown table, a missing warehouse, and a directory without a catalog (which stays empty). - Mutation checks: - dropping the read-only options fails the write test - dropping alias registration fails 8 tests - dropping the table listing fails the unknown-table test - reverting the catalog check to a directory check fails the empty-directory test - dropping `with_allow_ddl(false)` fails the external-table test - dropping `with_allow_dml(false)` fails the `EXPLAIN ANALYZE INSERT` test - enabling URL tables fails the file-path test - fmt, clippy, tests, typos, machete and `cargo doc` with `-D warnings` all pass. `cargo deny check` fails on RUSTSEC-2026-0459, which entered the RustSec database on 2026-11-09 and is not related to this change; #99 handles it on main. Workspace line coverage was 96.2% at the last CI run. --------- Signed-off-by: Mayur Kale <hello@mayurkale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
cargo deny checkfails on every branch since 2026-11-09, when RUSTSEC-2026-0459 (CVE-2026-119375) entered the RustSec database. SQLite before 3.54 can return uninitialised page-cache memory when a database is opened next to a crafted-walfile whose page size differs. Our copy is SQLite 3.46.0, bundled bylibsqlite3-sys 0.30.1, whichsqlx-sqlite 0.8.6pins andiceberg-catalog-sqlrequires (cargo tree -i libsqlite3-sys). The fix islibsqlite3-sys 0.39, and no sqlx release accepts it:sqlx-sqlite 0.9.0caps it below 0.38, and iceberg-rust main still requiressqlx 0.8.1. No dependency update clears the advisory today.This adds an ignore with a reason to
deny.toml, beside the three zstd ignores, with a comment saying when to remove it. Removal is tracked in #98.Exposure
Only
catalog.dbin the warehouse directory is opened through this SQLite, and Infostash is the only writer there. Triggering the bug needs a crafted file placed next to the catalog by someone who already has write access to that directory.About #71
crates/storagedeclaressqlxonly to turn on the SQLite driver of the copyiceberg-catalog-sqluses. #71 moves that declaration to sqlx 0.9, which builds a second sqlx and leaves the catalog's copy without a driver; its tests fail with a pool panic for that reason, and it would not clear this advisory anyway. #71 should wait for iceberg-rust to move to sqlx 0.9, or be closed with a dependabot ignore for sqlx majors.How it was tested
cargo deny checkpasses with the ignore and fails without it (one advisories error, RUSTSEC-2026-0459).typospasses. No code changes.