Skip to content

Latest commit

 

History

History
349 lines (316 loc) · 29.8 KB

verification.md

File metadata and controls

349 lines (316 loc) · 29.8 KB

Concurrent local verification and exact-input receipts

tauceti verify [--stage build|axioms|module-system|lint|shim-expiry|all] [--base SHA] [--json] [--no-reuse] [--lint-modules FILE] runs synchronously in a TauCeti checkout. It exits zero only after every requested phase succeeds against unchanged inputs. Progress, original tool output and actual exits go to stderr; --json writes one tauceti-verification/v1 object on stdout. The default authoring/repair prompts run the shared Tools local checks, also exposed by lake test and lake lint in current content checkouts. Receipt-helper dispatch remains opt-in. TAUCETI_VERIFY_HELPER=1 opts a selected natural canary into helper prompts; the helper CLI and receipt support are available separately. Helper prompts invoke the installed absolute scripts/tauceti-verify wrapper with the worker's Python interpreter. The wrapper also honors TAUCETI_VERIFY_PYTHON and supports Bubble's staged layout. Wheel entry points restore its execute permission, so login-shell PATH changes cannot hide it. Existing tauceti work, module and checkout entry points keep working.

The worker pins TAUCETI_VERIFY_BASE from the existing canonical-main checkout preparation, TAUCETI_VERIFY_POLICY from surveyed labels/branch, TAUCETI_VERIFY_STATE to worker state, and TAUCETI_ROUND_DEADLINE to the round's deadline. Standalone invocations resolve canonical main from local Git refs only. No scope or verification step requests GitHub data or fetches refs. The shared offline Python scope calculation is re-exported by verification_scope.py: merge-base diff, both rename ends, staged/dirty/deleted/untracked sources, ignored sources under TauCeti, full CI requirements for broad policy/pins/deletions/unrepresentable modules, and unknown scope on missing history. Missing canonical main fails closed with an incomplete receipt: there is no resolved base to bind evidence to. Unknown lint scope remains incomplete instead of silently establishing lint evidence. A stage-only build or audit does not claim the skipped requested lint succeeded.

All stages run the whole-library lake build --iofail, fast changed-file axiom diagnostics, module-system, source style lint, changed-module environment/documentation lint, and the shim ratchet. A stage-only call runs that stage; it does not assert that prerequisites or other stages passed. The shim checker receives merge-base source and registry snapshots, strict --fail-on-available, and --only-new for unchanged pins outside bump policy. Older source revisions without the checker record an empty shim phase. CI still owns whole-library environment lint. For failures in unchanged modules, --lint-modules /absolute/list adds validated existing dotted TauCeti module names to the local lint list; its contents are fingerprinted. Use the same option for final verification. Write this list and any extra diagnostic logs outside the checkout; untracked checkout files are inputs.

The shim wrapper runs with the verifier's Python interpreter and inherited environment, including Bubble's mounted Tools package and standalone checkers' sibling imports. Its installed TauCetiTools implementation and configuration are verification inputs. Local helpers use the Worker's installed Tools pin; checkout launchers and CI use the aligned content pin. Historical checkouts may retain older pins. A missing required Tools import yields incomplete_reason: tool-missing, rather than a source failure. Other checker exits retain their existing failure meaning. Recovery probes this prerequisite before claiming work or warming caches, so an unavailable Python dependency does not launch source repair. A changed verification-tool identity permits a fresh mechanical check of prior source failures, including records blocked only on continuation admission; publication scope, source ownership and model authorization still apply.

Local environment lint uses Tools' shared LocalLint.lean checker, compiled once and shared across workers with the same toolchain and dependency content. It imports only the explicit module list and checks declarations owned by those modules, including documentation and nolint entries. Imported dependency declarations are excluded. Empty or theorem-only documentation scopes are valid. Older PR scripts cannot restore a whole-project documentation scan: local verification never invokes their lint-env.sh. CI retains its full-library coverage.

Local axiom checks use Tools' shared LocalAxioms.lean, through the absolute scripts/tauceti-local-axioms --modules FILE wrapper. They read only the selected modules' own .olean, .olean.server and .olean.private data, without loading a Lean environment, following imports, reading Mathlib artifacts or running candidate initializers. They reject axiom declarations and direct references to sorryAx, Lean.ofReduceBool, Lean.ofReduceNat and Lean.trustCompiler, including private/opaque proof bodies. Missing/unreadable artifacts fail, an empty docs/deletion-only list is valid, and unknown scope fails instead of loading the library. Even bump/full-lint/deletion scopes check only their listed surviving changed modules locally. These are fast diagnostics after a kernel-checked build, not a transitive axiom or global declaration-ownership certificate. Those checks, including downstream importers and changed dependencies, remain in authoritative CI. lake exe axioms is not used locally: its filter only narrows traversal after a full environment import. Policy version 4 and the bundled driver/wrapper hashes invalidate earlier local verification receipts. The wrapper compiles a small native executable once per checker/toolchain identity, under a flock and atomic publication, then executes it directly. There is no Lake or Lean interpreter startup on the warm path, no result caching, and no interpreter or full-library fallback. Host binaries live in an owned private ~/.cache/tauceti/local-axioms directory in the login user's home (TAUCETI_LOCAL_AXIOMS_CACHE overrides it, isolated HOME does not); Bubble uses its own persistent per-worker cache, never the host's executable cache. Interrupted compilation leaves no published executable and retries automatically. Four recent native versions are retained. Empty scopes need no toolchain. Local axiom and module-system phases always rerun against the current artifacts, because a successful stage-only check before a build cannot certify the outputs of that later build. Compilation/tool failures are incomplete verification, distinct from a found axiom violation.

Each invocation atomically writes state/<worker>/verification/<id>/receipt.json before launching anything and updates it through completion. File and directory fsync precede publication. The receipt records requested phases, scope, start/end timestamps, tool exits, best-effort cleanup and full log paths. Reused phases refer to their original receipt. Receipts and logs are retained on failure or interruption. SIGKILL leaves the last durable receipt incomplete, even if a tool appeared to finish. No background completion is assumed.

Evidence is reusable by phase only after that phase succeeds and its before/after inputs validate against the same digest. Each newly completed phase is indexed before continuing: an interruption in a later phase preserves earlier evidence without claiming the entire invocation succeeded. The digest covers the full Git source inventory and actual bytes (including missing, dirty, renamed, untracked and ignored TauCeti sources), base SHA, dependency sources, configuration, environment value hashes, installed Lean distribution, executable hashes and versions, Python runtime, and worker verifier code/policy. Sensitive environment values are not written. Changes conservatively invalidate all phases. ProofWidgets' generated widget/package.json.hash and widget/package-lock.json.hash are excluded from dependency inventories; the JSON source files remain checked. Both normal rounds and recovery verification hydrate the shared expanded-artifact cache before verification starts. Complete inventories are stored once under verification-inputs/<digest>.json, referenced from receipts. A phase index avoids scanning old receipts. Per-file digests are cached by path, device, inode, size, mtime and ctime; any metadata change rereads the bytes. The metadata cache is bounded. Receipts and logs remain retained; there is no automatic deletion of incomplete evidence. After an immutable identity validates, inventory-only hashes for it are omitted from the persisted verification-hashes-v2.json cache. Individually fingerprinted executable hashes remain cached even inside that installation. Directory paths are stored once; mutable input records remain bounded at 200,000. Persisted directory recency and priority for currently used keys keep active hashes ahead of cold history. Cache saves merge only touched keys under a bounded one-second per-state lock; an unavailable cache cannot change the verification result or skip runtime cleanup. Migration imports matching V1 stat-keyed hashes without rereading their payloads until each workspace completes preparation. Early deadlines and isolated continuations retain migration eligibility for the original workspace. Completion markers live separately in verification-hash-roots/, with one exact lookup per workspace and no history scan. Warm preparation in migrated workspaces reads only V2, including after source changes. Fresh isolated candidates skip opaque V1 history with nonmatching paths/inodes. Repeated continuations reuse their V2 hashes. A toolchain switch may require one new payload hash if its stat key is absent from V2, even when an old V1 file has that key. The V1 file remains intact for rollback and may be removed once rollback support is no longer needed. Immutable identity files and receipt indexes keep their existing formats. The warm bound assumes a matching validated installation identity. A new identity after a reboot, mount/provenance change or identity-cache cleanup may require a full cold payload hash under the installation lock; other callers reuse the resulting seal. Writable/unavailable-cache fallbacks retain ordinary stat-keyed hashes and complete input validation. Temporary shim archives are removed after use. Source metadata sampling every two seconds catches mutations restored to their original bytes; a final full digest check rejects changed inputs. Mutable toolchain/dependency metadata is checked at phase boundaries rather than repeatedly scanning large installations during tool execution. Environment hashing is limited to build configuration and tool-search variables; provider credentials and agent-session variables are excluded. Generated Lake outputs are not source inputs. These fingerprints identify built inputs; they do not establish proof correctness or make writable caches or local receipts a security boundary. Verification never weakens the axiom checks.

Independent workers run build, axiom/duplicate audit, module-system audit and lint commands concurrently. Verification adds no host-wide slot limit, ticket scheduler, polling wait or capacity-specific retry/accounting path. Existing worker/fleet admission and host CPU/memory limits continue to apply. Per-job leases prevent duplicate execution of the same candidate; they never serialize independent candidates. The existing source/import guards and new-file length check run before the default build. Source style and environment lint run after their required artifacts exist. These checks use the shared Tools implementation; no GitHub request is added to local scope or lint.

TAUCETI_VERIFY_CACHE_DIR selects the shared immutable-installation identity cache. The default uses the host boot/machine identity and UID under /tmp. The directory must be owned by the user and not writable by others. Bubble mounts it at /opt/verification-cache and persists receipts separately under worker state. Cache locks protect creation/validation of one installation identity; they are released before expensive verification commands run. Unregistered writable installations retain complete inventory validation without requiring a shared cache. If the shared cache is unavailable or unsafe, host verification falls back to complete validation and Bubble uses a private cache directory. Cache availability never grants success or prevents the checks from running.

TAUCETI_VERIFY_CHECKPOINT_MARGIN reserves 600 seconds before the round deadline by default (a nonnegative number of seconds; fixtures explicitly use zero). At that boundary the helper returns incomplete with “checkpoint now”. Bubble candidate edits are saved as a base-bound binary patch and an untracked-file archive in verification-checkpoints/ before checks begin, including when there is insufficient time to start verification. These artifacts are retained alongside a durable verification job in verification-pending/. Interrupted candidates are copied to independent verification-candidates/ workspaces; disposable Bubble candidates are preserved before a non-reused phase starts. A fully reused invocation copies no dependencies or Git history. Copies preserve actual sources/index, local Git history and warm Lake/Mathlib build caches. The original Git index is read through a private copy. Linux uses independent CoW extents when supported, with portable copying otherwise; trusted external immutable host dependencies can retain their validated read-only identity. Absolute/escaping source symlinks fail incomplete because they need specific recovery ownership. Copying still has production storage/I/O cost to measure before broad dispatch. No clone contacts a remote. Candidate source/checkpoint data is retained for publication/reconciliation; disk retention and package history cost remain canary measurements.

Interrupted CLI results remain status: incomplete, exit nonzero, and include a typed incomplete reason, result_type: pending, the continuation job and its next automatic action. SIGKILL leaves the published job eligible once its live flocks release. The ordinary supervisor checks fleet/CI admission, then launches an owned fresh installed helper before model selection or GitHub reads. The helper inherits the job lease, works only in the isolated candidate, and does not hold the worker's round lock while building. Independent verification leaves inference scheduling available. One continuation per worker bounds preparation concurrency. Ordinary child polling reaps results; supervisor shutdown terminates its owned helper, preserving a durable interrupted job.

Continuation runtime status is separate from model-round status, so concurrent replay cannot overwrite authoring progress. Repeated deadlines retain the candidate and validated phase evidence. Phase boundaries finish metadata/source validation and index publication even if the checkpoint margin arrives at command completion. Complete input/tool-version preparation runs at invocation boundaries; phase checkpoints use existing input metadata and exact source/configuration checks. Job/index leases suppress duplicate exact-candidate jobs. Permanent scope/configuration/ownership errors become terminal incomplete results with retained artifacts, so they cannot starve inference. Missing tools remain dormant without repeated verifier launches until local toolchain availability, provenance or any installed verifier component changes. Toolchain probes retain separate stdout/stderr logs. Their final version/prefix lines bind the inputs; changing Lake package diagnostics are retained in logs and do not invalidate successful checks. A prefix must name an absolute installed directory containing bin/lean. Invalid final output still leaves verification incomplete. Deadline/signal jobs remain eligible without an expiry that discards their candidate. Terminal jobs move from verification-pending/ into verification-jobs/.

Candidate storage and automatic reclamation

Verification records host-local storage observations across worker HOME overrides and runtime versions. Bubble helpers mount the same ledger and record canonical host state paths. Logical-byte settings are cleanup targets: they do not reserve growth, gate admission, interrupt tools, refuse source preservation or park a verification job. Active work may exceed these targets while physical space is available. There is no fixed heavy-work queue or storage-based inference admission limit.

Default settings (positive integers except the percentage, which permits zero):

Environment variable Default Meaning
TAUCETI_VERIFY_STORAGE_BYTES 128 GiB Host/user target for eligible idle candidate data; triggers completed-cache reclamation
TAUCETI_VERIFY_CANDIDATE_BYTES 32 GiB Candidate-size target that triggers early completed-cache reclamation
TAUCETI_VERIFY_OWNERSHIP_SECONDS 2 Separate bound for one process-reference observation per cleanup slice
TAUCETI_VERIFY_MIN_FREE_BYTES 20 GiB Actual minimum filesystem headroom
TAUCETI_VERIFY_MIN_FREE_PERCENT 10 Percentage headroom capped at the worker critical reserve (at most 20 GiB); explicit MIN_FREE_BYTES remains a minimum
TAUCETI_VERIFY_COMPLETED_CACHE_SECONDS 900 Completed-job cache grace period (15 minutes)
TAUCETI_VERIFY_IDLE_CACHE_SECONDS 21600 Idle pending-job cache lifetime (6 hours)
TAUCETI_VERIFY_STORAGE_DIR Host-local state directory Shared observation root; use private state for tests

The former TAUCETI_VERIFY_SOURCE_CANDIDATE_BYTES, TAUCETI_VERIFY_CHECKPOINT_BYTES and TAUCETI_VERIFY_CHECKPOINT_POOL_BYTES caps no longer restrict preservation. They are accepted for configuration compatibility. Existing reserve helper calls register activity without preallocating 32 GiB. Twenty active candidates therefore do not book 640 GiB of imaginary future writes. Copies retain warm build outputs even when larger than the candidate target; source checkpoints can exceed the old archive/pool caps. Pending jobs blocked by an earlier storage limit resume mechanically without requiring a configuration change or a new model invocation.

Observations count logical bytes, including reflinked payloads, with a 16 KiB accounting floor. The persisted reserved/reserved_bytes fields retain their names for older readers; new activity records contain measured bytes or small metadata floors rather than speculative growth. Prepared status includes target_bytes and admission_limited: false. Newly discovered unmeasured trees start at a small metadata floor; bounded read-only inventories replace this with measured sizes before their caches are considered for expiry. Historical speculative growth estimates are excluded from cleanup pressure, so imaginary reservations cannot force needless cache eviction or block verification. A corrupt disposable ledger is quarantined and reconstructed locally without deleting source or evidence. Optional observation errors cannot invalidate verification or prevent receipt publication. Successful and interrupted invocations mark their actual idle time without a full tree walk on exit; bounded housekeeping owns complete size accounting. Invalid cleanup targets/TTLs use defaults and prepared warnings; invalid physical-headroom settings still require correction.

The ordinary supervisor, activity preparer and explicit tauceti cleanup invoke bounded reclamation. At most one collector runs per host, with a ten-second minimum interval; actual cadence depends on ordinary activity/round boundaries. The default deletion slice is 0.25 seconds, 2048 metadata entries and 256 MiB of removal. One process-reference observation has its separate two-second bound, so it cannot repeatedly consume the deletion allowance. Prepared metrics report that overhead and total wall time separately. Recursive removal has a durable journal; partial removal stays with the oldest progressing cache until complete, while blocked owners are skipped fairly. Its continuation is recovered even if a collector dies before saving the scheduling cursor. Stale inventory generations release their bounded scan slots when a candidate restarts or disappears. Eligible terminal or urgent outputs can be reclaimed before a complete source-tree size measurement. Inventories resume within a long-lived collector and restart safely after process death. They retain no directory descriptors between ticks that could masquerade as live workspace users. Cursor rotation prevents a live or large candidate starving others. Collection skips compacted rows, never follows symlinks and never crosses observed mounts. Job/receipt/workspace flocks and live process references protect active work; missing ownership or mount observations withhold that tree.

Completed candidates can shed their own and dependency build outputs after 15 minutes. Idle pending candidates normally shed only their own build outputs after 6 hours, measured from when an attempt stops. Logical targets cannot evict recent pending progress seconds before mechanical continuation. Completed caches can expire earlier above a size target. Each cleanup sweep starts with completed, oldest-idle candidates, rotates past blocked owners, and stops size-triggered eviction once enough has been reclaimed. Active candidates, checkpoints, recent pending progress and already compacted source-only trees do not consume the reclaimable-data target or force other caches to be flushed. Only actual low disk space overrides pending idle protection. As a last resort under that physical pressure, other idle jobs can shed dependency builds while the oldest runnable candidate's dependencies stay warm, avoiding repeated self-eviction/rebuild loops. Before removing a build directory the collector checks for tracked and untracked non-ignored files. Source under build/, staged state, dependency edits, Git history, checkpoints, receipts and publication/accounting journals remain protected. No unpublished source expires merely because it is old. If only active work or protected source remains above a target, status reports that fact and verification continues; a target is not a hard disk cap.

Failed copies carry a durable construction marker until isolated ownership is published. Exceptions remove partial copies and process death leaves bounded mechanical cleanup. A direct invocation claims its workspace before such disposal is permitted. Removed artifacts release their accounting. A cache generation token invalidates build receipt reuse after eviction without historical receipt scans. Cleanup errors cannot terminate the supervisor. If ENOSPC prevents writing cleanup intent, the collector may discard optional V2 stat-hash metadata under its writer lock and retry once. This can require cold hashing next time; ordinary warm operation retains the hash cache and receipt index.

Running tools sample actual free space once per second and inspect at most 4096 metadata entries per sample for cleanup observations. Only actual low filesystem headroom can interrupt their owned process group and return incomplete_reason: disk-pressure. An independent candidate remains pending for mechanical continuation once space returns, without a model or an attempt charge for admission refusal. If preservation itself encounters low disk space, the original workspace remains intact; without an independent candidate the CLI reports incomplete with accurate workspace retry guidance. Ownership/cleanup lease contention is separately typed verification-busy, never reported as logical capacity or actual disk pressure. A workspace with partially removed outputs stays owned by mechanical cleanup until that journal is complete; ordinary supervisor ticks finish it without a model or an admission-attempt charge. Other candidates remain eligible. Pending checkpoint mtime supplements the idle clock after abnormal exit or unavailable optional accounting. Source checkpoint writes also check actual headroom. Samples cannot prevent a fast writer exhausting space between observations; the targets are not filesystem quotas. This change does not raise capacity, restart active rounds or change V0 helper dispatch.

Recovery builds only in an independent candidate, never resets the original source and never pushes, creates a PR or bypasses accounting/publication guards. Jobs store canonical candidate paths. The helper resolves runtime state/workspace symlinks before checking that its working directory matches the job and validating the inherited job lease. Receipts remain bound to that physical workspace, including when a runtime alias changes after deployment. Receipts bind the actual replay environment; a different toolchain/configuration invalidates prior evidence. Bubble recovery needs the host's normal verification tools for the pinned project; unavailable tools leave typed incomplete evidence and a dormant job that wakes on local tool availability changes. Publication/reconciliation of a recovered candidate remains #310's separate concern. This verification continuation does not require #310. Claude Bash timeout variables cover the remaining round; Codex prompts instruct the agent to wait on its returned execution session. SIGINT, SIGTERM and SIGHUP cause owned cleanup and incomplete receipts.

Tools run in owned process groups. Interruption and deadlines terminate the group, and normal completion sweeps leftover background tools. Cleanup is best effort: deliberately detached descendants can escape. Host verification needs no rootless PID namespace or historical cleanup proof. Explicit Bubble execution retains its existing outer sandbox. Normal exits check each command directly; shell lint drivers use Bash pipefail. No process-name polling is used. Job/invocation lease descriptors are never inherited by tools, so owner death releases ownership for recovery. Recovery verifies in an independent workspace; publication retains its separate source, claim and CAS checks.

TAUCETI_VERIFY_IMMUTABLE_INSTALLATIONS=/absolute/installations.json can register trusted installations with a mapping such as {"/nix/store/example-lean": "publisher-revision-1"}. Paths must be fully resolved. The publisher must make installations immutable, including writable aliases, and replace the installation/provenance token on updates. Registration alone grants no fast path: Linux must report a read-only filesystem, all nested mounts must be read-only, and cold validation rejects external symlink targets. Unsupported hosts and writable/unregistered installations retain complete inventory and stat-keyed hash validation. A cold validation stores a content digest and file count in a host-shared cache under the private verification cache directory, serialized per identity. Separate worker states reuse that validated installation. Warm validation binds that digest to the root device/inode/metadata, mount topology and publisher provenance; it performs no recursive immutable walk. A currently writable mount, replacement root, changed mount topology or changed provenance invalidates the identity, including at phase boundaries. A writable-to-read-only remount cycle that restores the observed topology is not detectable: the trusted publisher must change provenance after any mutation, including writes through aliases. Read-only mount observations alone do not establish immutability. Source/configuration, base, executable/version and verifier checks remain exact. The trust contract and local cache are accidental-mutation guards, not protection from an adversary changing read-only aliases or receipts.

Set TAUCETI_VERIFY_HELPER=0 to retain/restore bundled V0 prompts at the next round. This changes dispatch only: it does not delete receipts/checkpoints/jobs or terminate active verification. TAUCETI_VERIFY_CONTINUATION=0 separately pauses launching new mechanical continuations while retaining eligible jobs and allowing active helpers to finish. Green fixture tests do not authorize blanket fleet dispatch.

Implementation evidence

tests/verification.py covers real exits, pipefail, exact and phase reuse, source mutations, pin/base/toolchain/executable changes, deletions, renames, untracked/ignored sources, owner death, deadlines and owned-group cleanup. Twenty independent fake verifiers must enter their builds together, even with an old one-slot lock held and stale wait records present. tests/verification_audit.py adds repeated deadlines followed by an actual successful build in the isolated candidate, Bubble namespace recovery after original deletion, phase evidence after interruption, dispatch gating and immutable preparation costs. Its 30,000-file, 122,880,000-byte installation uses fixture read-only statvfs flags with the real mountinfo reader/parser; real writable installations are also tested to fall back. Twenty contenders use separate worker states and the host-shared validated identity cache. It prints cold and twenty-contender warm bytes, file records, stat/lstat calls and walks, with 2,000 unrelated historical receipts. A local run observed 122,880,000 bytes, 30,000 file records and 180,204 stat/lstat calls cold; twenty warm contenders performed zero payload reads/file records/walks and 100 stat/lstat calls, plus 40 statvfs calls and 40 mountinfo reads (108,680 bytes on that host). This fixed mount-metadata cost depends on host mount topology. These are fixture preparation costs, not a production throughput measurement. Warm payload bytes/file records/walks must be zero, and metadata work remains bounded independently of immutable payload/history. All fixtures use private activity/cache/state paths and fake verifiers; they launch no paid agents.

Deployment evidence

No fleet dispatch, capacity, spending or active-session changes are made by these fixtures. V0 remains the default. Opt-in natural canary measurements of useful completions, copying cost and host concurrency are still required before broad dispatch.

Natural recovery evidence

No natural production recovery or numerical throughput improvement is asserted by local fixture results. Record observed candidate/job/receipt IDs, next automatic action, completed useful work, verification concurrency and immutable preparation cost separately when a canary naturally selects work.