Add bounded native SQLite inspection across model tools, CLI and MCP

Author: ShadowfetchStudio

CHANGELOG.md

@@ -12,6 +12,13 @@
   existing profiles, without deleting data or changing existing parent-directory
   permissions. Unsafe profile leaves and lock files are rejected. Notifications
   keep full text contrast throughout their entrance animation.
+- Native SQLite tools, CLI and MCP queries replace the Python built-in reader,
+  with parameter binding, live WAL support, confined paths, denied SQL writes
+  and bounded/cancellable results. Bundled SQLite is updated to 3.53.2, including
+  its WAL-reset corruption fix.
+- Required task inspection recognizes successful native database reads. Small
+  model contexts can use a shorter, enforced response budget while retaining
+  required input; requests that still cannot fit fail before contacting the model.
 - Native goals preserve milestone results, pause and resume, require recorded
   inspection/verification for the default checklist, and stop on failed checks.
   Goal tasks appear live in the conversation; results and progress are accessible

Cargo.lock

@@ -19,6 +19,7 @@ workflows, [managed background processes](docs/NATIVE_BACKGROUND.md), and
 [reviewed lifecycle commands](docs/NATIVE_HOOKS.md),
 [project inspection and diagnostics](docs/NATIVE_INSPECTION.md),
 [project and task notes](docs/NATIVE_MEMORY.md),
+[native SQLite inspection](docs/NATIVE_SQLITE.md),
 [approved MCP stdio and HTTP tools](docs/NATIVE_MCP.md), a
 [native MCP stdio server](docs/NATIVE_MCP.md#connect-another-coding-tool-to-shadowcode), and a
 [native CLI](docs/NATIVE_CLI.md) that shares the active desktop engine or runs

README.md

@@ -63,6 +63,14 @@ Review applies only current unstaged hunks. A changed diff causes a conflict,
 rather than applying a stale client patch. File restores can overwrite later
 edits; review the checkpoint and retain independent version-control backups.
 
+[Native SQLite inspection](docs/NATIVE_SQLITE.md) opens existing project
+databases read-only and authorizes only queries returning rows. SQL writes,
+ATTACH, configuration PRAGMAs and extension loading are denied; a small explicit
+allowlist supports read-only schema PRAGMAs. SQLite may maintain its WAL
+coordination sidecars. Directory capabilities confine database/sidecar paths;
+query work, concurrent readers and output are bounded. Cancellation interrupts
+SQLite and releases its connection; an OS filesystem stall may delay return.
+
 ## Secrets and transcript content
 
 Keys belong in `~/.config/shadow-agent/secrets.env` with mode 600, or environment

SECURITY.md

@@ -41,6 +41,11 @@ uses an exact task in the current project. `--json` exposes content hashes;
 string clearing it. See [native memory](NATIVE_MEMORY.md) for persistence,
 legacy notes, continuation, branching and export.
 
+`sqlite PATH [SQL]` lists tables or reads an existing project database.
+`--params '[100]'` binds positional placeholders; `--limit 50` and
+`--timeout-ms 2000` reduce result/work limits. Check `truncated` in the response.
+See [SQLite inspection](NATIVE_SQLITE.md) for types, limits and WAL behavior.
+
 `hooks` lists project lifecycle command definitions. Review the command and its
 hash, then use `hooks --enable PATH --hash HASH` to enable those exact contents
 or `hooks --disable PATH` to remove the registration. Hooks remain inactive in

docs/NATIVE_CLI.md

@@ -62,6 +62,10 @@ the CLI/MCP server, with bounded continuation context, independent branch copies
 and note exports. Approval decisions carry the displayed prompt's conversation
 ID so a navigation change cannot send the decision to another task.
 
+[Native SQLite inspection](NATIVE_SQLITE.md) gives models read-only table/query
+tools, also available through the CLI and MCP server, with parameter binding,
+live WAL support, cancellation and result limits.
+
 Project [lifecycle commands](NATIVE_HOOKS.md) can be reviewed and enabled in
 Settings → Hooks or through the same native CLI. Their command gates and
 completion checks run inside the native task lifecycle without Python callbacks.

docs/NATIVE_DESKTOP.md

@@ -3,8 +3,8 @@
 MCP remains an open [native release gate](NATIVE_MIGRATION.md). The Rust stdio and Streamable HTTP
 clients now connect to native Settings, CLI registration, project activation,
 and individually approved agent tool calls. The native stdio server now exposes
-tasks, goals, reviews, memory and checkpoints to external clients. Built-in SQLite
-compatibility, the remaining server features, and broader interoperability/real-model
+tasks, goals, reviews, memory, SQLite inspection and checkpoints to external clients.
+The remaining server features and broader interoperability/real-model
 verification are still being migrated. Neither direction launches Python.
 
 ## Connect another coding tool to ShadowCode
@@ -46,7 +46,7 @@ client's task or all pending actions. Denying an owned approval does not require
 the approval flag. The client is responsible for obtaining the user's agreement
 to the displayed action before submitting approval.
 
-The current server exposes sixteen tools:
+The current server exposes seventeen tools:
 
 - `shadow_status`, `shadow_models`, `shadow_sessions`, `shadow_review`, and
   `shadow_tools` inspect the selected project and native capabilities.
@@ -57,6 +57,9 @@ The current server exposes sixteen tools:
 - `shadow_memory` reads, appends or replaces project/task notes. Task scope uses
   an exact existing task ID in this project; replacement requires the current
   content hash. Changes require write access. See [native memory](NATIVE_MEMORY.md).
+- `shadow_sqlite` lists tables or runs one bounded read-only query against an
+  existing project database. It supports bound parameters and live WAL data;
+  SQLite may maintain coordination sidecars. See [SQLite inspection](NATIVE_SQLITE.md).
 - `shadow_goal` creates, lists, inspects, advances or abandons durable goals.
   Creating or advancing a goal here does not start an agent automatically.
 - `shadow_run` creates a fresh conversation and returns its job ID. `shadow_jobs`

docs/NATIVE_MCP.md

@@ -153,10 +153,9 @@ transports, inert Settings/CLI registration, explicit project/content activation
 discovery, individually approved calls, and task-owned process/stream cleanup.
 HTTP uses explicit credential references, prevents redirects and automatic call
 retries, and supports modern and legacy Streamable HTTP negotiation. The native
-stdio server exposes sixteen tools, four resources and two prompts through
+stdio server exposes seventeen tools, four resources and two prompts through
 the shared engine, with a fixed project, reduced per-task permissions, explicit
-approval delegation and connection-owned jobs. Built-in SQLite compatibility,
-HTTP serving and broader interoperability/real-model
+approval delegation and connection-owned jobs. HTTP serving and broader interoperability/real-model
 verification remain open requirements. Private engine-side ownership now cancels
 MCP jobs after gateway SIGKILL, including unread submission replies, queued tasks
 and active command children; unrelated work continues.
@@ -172,6 +171,11 @@ notes, legacy-file preservation, atomic edits with stale-hash rejection, bounded
 continuation context, independent branch archives and complete note exports.
 The desktop, CLI and MCP server share this service.
 
+[Native SQLite inspection](NATIVE_SQLITE.md) replaces the Python built-in tools
+with confined, bounded read-only queries, typed results, parameter binding,
+live WAL coordination and cancellation. Model tools, CLI and MCP share the
+reader. The history store also uses the updated bundled SQLite 3.53.2 library.
+
 AppImage extraction now uses a source-pinned runtime patch with private
 per-invocation directories, signal forwarding, and checked cleanup. A dedicated
 regression exercises concurrent owners/clients, deferred WebKit resources,

docs/NATIVE_MIGRATION.md

@@ -10,6 +10,13 @@ a specific model in the composer overrides the route for that task. The model
 note above the response records the actual model, provider, and purpose, and
 survives reload. The context meter uses that task's model context limit.
 
+The engine normally reserves a quarter of that window for a response (up to
+8,192 tokens). If essential input leaves less room after history compaction,
+it can request a shorter response down to 256 tokens. The actual provider
+request uses that reduced limit; the configured context window never grows
+silently. If input, tools and the minimum response still cannot fit, the task
+reports the estimated requirement and stops before sending an oversized request.
+
 Default goal milestones use Plan for inspection, Build for implementation, and
 Test for required command verification. Planning and review retain read-only
 permissions regardless of the selected model. A different model never grants

docs/NATIVE_ROUTING.md

@@ -0,0 +1,93 @@
+# Native SQLite inspection — 0.20 development
+
+The native engine lists tables and queries existing SQLite files inside the
+selected project. It replaces the Python built-in reader with bundled SQLite;
+no Python, shell, database CLI or external MCP registration is required. The
+same reader serves model tools, the CLI, desktop IPC and the MCP server.
+
+## Use it
+
+```sh
+shadowcode --workspace /path/to/project sqlite app.db
+shadowcode --workspace /path/to/project --json sqlite app.db \
+  'SELECT id, name FROM users WHERE id > ? ORDER BY id' \
+  --params '[100]' --limit 50
+```
+
+With no SQL, the response includes `tables` and their CREATE TABLE definitions
+in `rows`. Queries return `columns`, `rows`,
+`truncated`, `limit`, and `read_only`. Check `truncated`; narrow the predicate
+or use an ordered keyset query to retrieve further rows. Duplicate column names
+are rejected: use `AS` aliases rather than losing a value in a JSON object.
+NULL, integers, finite numbers and UTF-8 text retain their types. BLOBs return
+`{ "type": "blob", "hex": "00ff", "bytes": 2 }`. Invalid UTF-8 and non-finite
+numbers fail explicitly; `hex(column)` can produce a readable representation.
+
+Models use the legacy names `mcp_sqlite_tables(path)` and
+`mcp_sqlite_query(path, sql, params?, limit?)`. They are native tools available
+in Plan/Review and Build, with durable task results and bounded parallel reads.
+External MCP activation is unrelated to these built-ins.
+
+External clients use `shadow_sqlite` with the same arguments, optionally adding
+`timeout_ms`. Omit `sql` to list tables. The default read-only server supports it;
+the selected project remains fixed. Application clients use `POST /api/sqlite`
+with that JSON body through the native service.
+
+## Data safety and live databases
+
+The main file opens read-only and must already exist. SQLite's authorizer
+denies data/schema writes, ATTACH/DETACH, configuration PRAGMAs, transactions,
+extension loading and file-access functions. A query must be a single statement returning columns;
+SELECT, WITH and subqueries work. Read-only metadata PRAGMAs are allowed:
+`table_info`, `table_xinfo`, `index_list`, `index_info`, `index_xinfo`,
+`foreign_key_list`, and `table_list`, including their table-valued forms.
+Parameters are bound separately from SQL.
+A bounded preparation pass rejects multiple statements without recursively
+walking an arbitrary chain of SQL statements.
+
+Live WAL reads retain SQLite's normal locking and transaction semantics and
+see committed WAL data; the connection never asserts `immutable=1`. SQLite may
+create or update its `-wal`/`-shm` coordination files, including after a writer
+closes or leaves WAL data without shared memory. This bookkeeping does not
+grant SQL write access or change database rows/schema. Tests check database and
+existing WAL bytes remain unchanged while reading a separate process's commits.
+An exclusive lock, inaccessible coordination file, corruption, or recovery that
+requires a database write produces an error rather than bypassing SQLite locks.
+
+The Linux reader holds the workspace and every real parent directory open. A
+private VFS preserves the directory capability's `/proc/self/fd` path while
+delegating locking and I/O to SQLite's bundled Unix VFS. Parent/final symlinks,
+special files and existing sidecars with additional hard links are rejected.
+This shares the application's same-account trust boundary; it is not an OS
+sandbox against another process controlling the user's files.
+
+## Limits and cancellation
+
+- 200 rows by default, maximum 1,000; serialized output is capped at 1 MB.
+- Five seconds by default; CLI `--timeout-ms` and API `timeout_ms` accept
+  1–10,000 ms. Model tools also respect their configured tool timeout.
+- Eight simultaneous readers per process; excess requests fail with a retry hint.
+- SQL and parameter data each have a 64 KB limit, with at most 128 parameters
+  and 128 result columns. SQLite values/rows are limited to 1 MB.
+- Ten million VM instructions, reduced parser/VM limits, an approximately 2 MB
+  configured page cache, and no auxiliary SQLite threads. Queries requiring
+  temporary disk spill fail instead of keeping an unbounded in-memory working
+  set or creating temporary files. Use indexed or narrower queries.
+
+Cancellation/deadlines use SQLite's interrupt handle and progress callback.
+Abandoning a request cancels its worker, which retains its reader permit until
+the connection closes. SQLite interruption is cooperative; a stalled kernel
+filesystem operation may delay return. Tests check that cancellation releases
+locks for a subsequent writer.
+
+The locked native library bundles SQLite 3.53.2, including the upstream
+[WAL-reset fix](https://www.sqlite.org/wal.html#walreset). This also updates
+ShadowCode's history store; schema 24 and existing migration backups remain.
+The implementation follows SQLite's [WAL documentation](https://www.sqlite.org/wal.html),
+[VFS contract](https://www.sqlite.org/c3ref/vfs.html), and
+[untrusted-SQL guidance](https://www.sqlite.org/security.html).
+
+The [verification record](NATIVE_VERIFICATION.md#real-local-models) includes real
+gpt-oss and Qwen runs that discover a disposable database's schema, execute the
+correct aggregate query and report its result without changing database bytes.
+Successful SQLite reads also count toward required task inspection.