Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions .claude/tasks/prod-readiness/README.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
== Production-Readiness — Execution Pack

Self-contained work orders from the 2026-07-10 ten-dimension
production-readiness audit of statistikles. Each file is a standalone
brief runnable by a fresh Claude session with *no dependency on the
originating conversation*, with a recommended model (*Opus / Sonnet /
Haiku*) for implementation and for adversarial verification.

* *Wave 1* = "`basically running & functioning acceptably`" (the P0
guarantee + the crash/robustness/coverage/install fixes). Do these
*first*.
* *Wave 2* = everything beyond that (release engineering, packaging,
honesty reframes, deeper test coverage, observability, polish).

All of wave 1 and most of wave 2 are now *merged to `+main+`*. Only two
work orders remain open: *W2-6 (observability)* and *W2-7
(prompt-injection delimiting)*.

=== How to run

*One task, any session:* > "`Execute the work order in
`+.claude/tasks/prod-readiness/w1-1-neural-guardrail.md+`. > Use the
model it names; follow the execution contract in the README.`"

*A whole wave as a fleet (Claude Code, budget permitting):* >
`+Workflow({ name: "prod-readiness" })+` — the runner reads this folder,
routes each > task to its model, and does implement → adversarial-verify
→ open-PR. See > `+.claude/workflows/prod-readiness.js+`.

=== Model routing

==== Wave 1 — basic functioning (do first, in order)

*Status (2026-07-11):* all eight wave-1 packages are *MERGED* to
`+main+`.

[width="100%",cols="10%,15%,20%,15%,20%,20%",options="header",]
|===
|# |Task |Branch |Impl |Verify |Status
|1 |link:w1-1-neural-guardrail.md[Neural boundary guardrail (*P0*)]
|`+fix/neural-boundary-guardrail+` |*opus* |*opus* |✅ #37

|2 |link:w1-2-stats-degenerate-inputs.md[Degenerate-input guards +
`+@assert+`→`+ArgumentError+`] |`+fix/stats-degenerate-inputs+` |sonnet
|opus |✅ #40

|3 |link:w1-3-executor-router-coverage.md[Table-driven router tests + CI
coverage] |`+test/executor-router-coverage+` |sonnet |sonnet |✅ #41

|4 |link:w1-4-documented-install-path.md[Real install path
(Justfile, quickstarts, smoke CI)]
|`+fix/documented-install-path+` |sonnet |haiku |✅ #34

|5 |link:w1-5-supply-chain-pinning.md[Pin compute half + prune
Dependabot + threat model] |`+fix/supply-chain-pinning+` |sonnet |sonnet
|✅ #35

|6 |link:w1-6-zig-ffi-compiles.md[Make Zig FFI compile +
`+zig build test+` CI] |`+fix/zig-ffi-compiles+` |*opus* |sonnet |✅ #38

|7 |link:w1-7-agda-proofs-ci.md[Type-checking Agda proofs +
`+agda --safe+` CI] |`+fix/agda-proofs-ci+` |*opus* |sonnet |✅ #42

|8 |Hygiene / security-templates |`+chore/hygiene-security-templates+`
|haiku |haiku |✅ #33
|===

==== Wave 2 — beyond basic functioning

*Status (2026-07-11):* six of eight merged; *W2-6* and *W2-7* remain
*OPEN*. A dedicated chi-square correctness review (Yates-clamp fix +
ground-truth tests) also landed as *#47*, outside the original wave
plan.

[width="100%",cols="10%,15%,20%,15%,20%,20%",options="header",]
|===
|# |Task |Branch |Impl |Verify |Status
|1 |link:w2-1-release-pipeline.md[Release: JuliaRegistrator + TagBot +
SBOM] |`+feat/release-registrator-tagbot+` |*opus* |*opus* |✅ #43

|2 |link:w2-2-guix-package.md[Buildable guix.scm]
|`+feat/guix-real-package+` |*opus* |sonnet |✅ #39

|3 |link:w2-3-containers.md[Runnable Containerfile + devcontainer]
|`+feat/containers-runnable+` |sonnet |sonnet |✅ #44

|4 |link:w2-4-experimental-reframe.md[Experimental reframe: FFI + proofs
docs] |`+docs/experimental-reframe+` |sonnet |sonnet |✅ #48

|5 |link:w2-5-reference-validation.md[Extend ground-truth reference
validation] |`+test/reference-validation-extension+` |sonnet |*opus* |✅
#46

|6 |link:w2-6-observability.md[Structured logging + audit trail]
|`+feat/structured-observability+` |sonnet |sonnet |*OPEN*

|7 |link:w2-7-prompt-injection.md[Prompt-injection delimiting]
|`+fix/prompt-injection-delimiting+` |sonnet |haiku |*OPEN*

|8 |link:w2-8-polish-sweep.md[Polish sweep] |`+chore/polish-sweep+`
|haiku |haiku |✅ #45
|===

*Routing rationale.* Opus for design-sensitive, safety-critical, or
niche-toolchain work (the guarantee guardrail, release engineering,
Guix, Zig/Agda, statistical ground-truth). Sonnet for well-specified
implementation. Haiku for mechanical sweeps. Verification is adversarial
— the verifier tries to _refute_ the implementation against the work
order’s acceptance criteria — and is deliberately assigned the model
that should be able to follow the spec: if the verify-model can’t
confirm it, the spec or the code isn’t done.

=== Decisions of record (user-approved 2026-07-10 — do not relitigate)

[arabic]
. *Release = JuliaRegistrator + TagBot* (General registry), not
artifact-only.
. *FFI (Zig/C-ABI) and Agda proofs are EXPERIMENTAL* — make them compile
& CI-check (wave 1), reframe docs (wave 2); do NOT invest in the Idris2
ABI or proofs-over-ℝ yet.
. *guix.scm gets made real* (buildable), not deleted — governance has a
"`Guix primary`" policy check.
. *Merge gate is active*: the Base ruleset requires the "`E2E — Julia
Test Suite`" status check. A PR cannot merge red.

=== Verified toolchain facts (WSL Debian, set up 2026-07-10)

* *Julia 1.10.11* via juliaup; on PATH only in a *login shell*
(`+bash -lc+`). Baseline `+Pkg.test()+` = *4404 tests green* (Full 424 /
E2E 145 / Property 3800 / Reference 35). ALWAYS serialize Julia runs:
`+flock /tmp/statistikles-julia.lock -c '…'+` (16 GB RAM ceiling). Warm
depot exists; `+instantiate+precompile+` ≈ 1m46s, test ≈ 36s after
precompile.
* *Zig 0.16.0* tarball at
`+/home/hyperpolymath/zig/zig-x86_64-linux-0.16.0/zig+` (also
`+~/.local/bin/zig+` in login shells). ⚠ 0.15+/0.16 has breaking std
changes (Io writer/reader redesign) — if the repo’s Zig targets ≤0.14,
fetch an older tarball into the same dir.
* *Agda 2.6.4.3 + agda-stdlib 2.1* (apt); stdlib wired via
`+~/.agda/libraries+` + `+~/.agda/defaults+`. ⚠ stdlib 2.1 renamed some
modules vs 1.x — expect import tweaks.

=== Execution contract (every task)

* Branch from up-to-date `+origin/main+` using the named branch. If a
wave-1 prerequisite is unmerged and you touch the same files, branch
from that branch and say so in the PR.
* Commits SSH-signed (environment is pre-configured). End messages with
the `+Co-Authored-By+` trailer for your model.
* Run the work order’s *Local verification* before pushing; on 16 GB
RAM, wrap Julia in `+flock /tmp/statistikles-julia.lock+`. Never claim
verification you didn’t run.
* Diffs surgical; match surrounding style. GitHub Actions *SHA-pinned*
with a version comment. Never job-level `+hashFiles()+`/`+secrets+`
conditionals (silent startup_failure) — step-level only.
* *Do not open GitHub issues* and *do not merge/delete/force-push.* Open
exactly one PR to `+main+` per task, titled per the work order; body =
what changed & why (file refs) + verification run & result + anything
skipped. Stop there.
* Python is banned in repo code (governance-enforced). Deriving test
constants with Python/R locally is fine; only the constants + a
derivation note enter the repo.
112 changes: 0 additions & 112 deletions .claude/tasks/prod-readiness/README.md

This file was deleted.

96 changes: 96 additions & 0 deletions .claude/tasks/prod-readiness/w1-1-neural-guardrail.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
== W1-1 · Neural boundary guardrail (P0) + hardening

*Model:* impl=opus · verify=opus · *Branch:*
`+fix/neural-boundary-guardrail+` *Priority: P0 — this is the product’s
raison d’être made real.*

=== Context

The flagship guarantee — "`no number is ever produced by the LLM`" — is
currently *prompt-only* (`+src/tools/chat.jl+` SYSTEM_PROMPT ~lines
29-66) with zero output validation. `+chat.jl+` (~114-119) prints
assistant content verbatim; nothing checks that numerals in the reply
came from a tool result. Surrounding this are crash/robustness gaps:
unguarded tool-call JSON parsing, no HTTP timeout, silent-null unknown
sub-types, dropped `+tools+` on follow-up calls (no multi-step
chaining), and `+@assert+`-based resource assumptions.

=== Requirements (implement in `+src/tools/+`)

*(a) NEW `+src/tools/guardrail.jl+`:* - `+collect_numbers(x)+` —
recursively harvest every numeric value from tool-result
Dicts/Vectors/nested structures. -
`+extract_numeric_tokens(text::String)+` — find numeric literals in
assistant prose (ints, decimals, scientific notation, percentages). -
`+validate_numeric_provenance(text, tool_results, user_numbers; rtol=1e-6) -> (ok::Bool, orphans::Vector{String})+`.
A token is legitimate if it approx-matches (rtol) any harvested
tool-result number, OR its ÷100 / ×100 variant matches (percent
phrasing), OR it appears in the user’s own input numbers, OR it is a
small structural integer 0..12.

*(b) `+chat.jl+`:* record all tool results for the turn; after the final
assistant content, run the guardrail (parse user-message numbers as
`+user_numbers+`). If orphans exist AND tool calls happened: ONE retry
asking the model to restate using only tool-result numbers; if orphans
persist, print the reply with a clear warning block listing the
unverified numbers. If the reply has numeric tokens but NO tool call was
made: same retry-once-then-warn with a "`no symbolic computation was
performed`" message. *Never silently rewrite model text — flag, never
fabricate.*

*(c) `+lmstudio.jl+` `+process_tool_calls+`:* wrap the per-tool-call
body (nested key access + `+JSON3.read+` of arguments, ~lines 88-101) in
try/catch; on failure push a `+role:"tool"+` message with a clean
`+Dict("error"=>...)+` so the model recovers. *Pass the `+tools+`
parameter on the follow-up call* (currently dropped ~line 104) and
iterate tool-call rounds in a bounded loop (max 5) until a reply has no
tool_calls. Add HTTP timeouts to `+call_lm_studio+` and the `+chat.jl+`
HTTP call: `+connect_timeout=10+`, `+readtimeout=120+`, `+retry=false+`
(see `+echidna_adapter.jl+` for the existing pattern); timeout → return
the existing error-Dict shape.

*(d) `+chat.jl+` REPL while-loop:* try/catch around the turn body —
print a concise error and continue; one bad turn must never kill the
session.

*(e) `+executor.jl+`:* clamp caller-supplied
`+n_reps+`/`+n_permutations+` to ≤100_000 and component counts `+k+` to
≤20 (return `+Dict("error"=>...)+` when exceeded); add a trailing
`+else return Dict("error"=>"Unknown type '…' for <tool>")+` to EVERY
inner sub-type dispatch (grep every inner if/elseif chain: t_test,
time_series, information_theory, survival_analysis, robust_stats,
causal_inference, spatial_stats, advanced_modeling, algebraic_stats,
nonparametric_test, and any others); gate the `+trace+` backtrace field
in the catch-all behind ENV `+STATISTIKLES_DEBUG+` (default off),
keeping a concise stable error string.

*(f) tests — NEW `+test/guardrail_test.jl+`:* unit tests for the three
guardrail functions incl. a *clean fixture* (all numbers from tool
results, with rounding/percent variants) that passes and an
*injected-fabrication fixture* that MUST be flagged; tests for
`+process_tool_calls+` recovery with malformed tool_call dicts (missing
keys, non-JSON arguments) — construct response Dicts directly, no HTTP;
tests for clamps and unknown-sub-type else-errors via `+execute_tool+`.
Wire into `+test/runtests.jl+`.

=== Acceptance criteria

* [ ] Guardrail flags an injected fabricated number and passes a clean
reply (both tested).
* [ ] Malformed tool-call dicts recover instead of crashing (tested).
* [ ] Every inner sub-type dispatch has an `+else+`-error; clamps
enforced (tested).
* [ ] HTTP calls have timeouts; follow-up call passes `+tools+`; loop is
bounded.
* [ ] Full suite green locally (was 4404 tests) + the new tests.

=== Local verification

`+flock /tmp/statistikles-julia.lock -c 'cd <repo> && julia --project=. -e "using Pkg; Pkg.test()"'+`
in a WSL login shell
(`+wsl.exe -d Debian -u hyperpolymath -- bash -lc '…'+`).

=== Out of scope

Prompt-injection input delimiting (that is W2-7, which builds on this
guardrail).
Loading
Loading