V2.10.S11: Reduce Test Memory Footprint

[deeprnd]

Product Outcome

Developers with consumer-grade hardware (16 GB RAM or less) can run Firedancer
unit tests (test-unit-fd) and system model tests (test-system-tk) without
OOM kills, page-swap thrashing, or manual memory overrides. The Firedancer test
launcher defaults to a 1 GB per-job workspace budget instead of the previous
implicit default; test-system-tk loads a 1–2 GB model instead of the ~16 GB
default Gemma 4 Q4_K_XL, cutting live-LLM memory pressure by ~85 %.

User Story

As a developer-operator on consumer hardware, I want test commands to fit within
a reasonable memory budget so that I can run the full test suite without running
out of RAM, enabling local iteration without a dedicated build machine.

As a CI runner, I want predictable memory usage so that test jobs do not consume
more resources than necessary, reducing CI costs and avoiding out-of-memory
cancellations.

Scope

• In scope: reduce test-unit-fd per-job workspace budget to 1 GB
  (--job-mem 1073741824) in the Firedancer test launcher.
• In scope: replace the test-system-tk default model file with a 1–2 GB
  quantized model (e.g. Qwen2.5-1.5B or Qwen2.5-3B in Q4_K_M GGUF).
• In scope: update environment defaults (ensure_hf_model.sh,
  run_llm_server.sh) so existing CI and local setups pick up the smaller model
  without requiring env var overrides.
• In scope: verify just test-unit-fd and just test-system-tk pass with the
  new defaults.
• Out of scope: Firedancer integration-test or e2e memory changes.
• Out of scope: changes to Firedancer tile behavior, policy, audit, or financial
  logic.
• Out of scope: changes to Tickoni runtime, capability envelopes, or model
  gateway production configuration.

Preconditions And Assumptions

• The Firedancer unit test launcher (contrib/test/run_unit_tests.sh) defaults
  PAGE_CNT by computing JOB_MEM / page_size. The default JOB_MEM is
  1 GiB (262144 normal pages). The current test-unit-fd justfile recipe
  passes --page-sz normal but relies on the default JOB_MEM. Reducing
  JOB_MEM to 1 GiB makes the default explicit and matches the current default.
• The current test-system-tk model is
  unsloth/gemma-4-E2B-it-qat-GGUF/gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
  (~16 GB disk, ~16 GB GPU/CPU RAM at inference). A 1–2 GB model such as
  Qwen/Qwen2.5-1.5B-Instruct-GGUF (Q4_K_M ~1.0 GB) or
  Qwen/Qwen2.5-3B-Instruct-GGUF (Q4_K_M ~2.0 GB) fits the target budget.
• The test uses fixture mode when no LLM endpoint is reachable, so unit-level
  behavior is unchanged. Only the live system test path requires the model.
• The smaller model must still pass the investment conformance test in
  contrib/test/run_system_model_tests.sh — this is verified as an acceptance
  criterion.

Acceptance Criteria

• Given just test-unit-fd runs on Linux x86_64, then the Firedancer unit
  test launcher uses a per-job workspace budget of 1 GB (--job-mem 1073741824, or --page-sz normal --page-cnt 262144), and all unit tests
  pass without OOM kills.
• Given just test-system-tk runs on Linux x86_64 with a local llama.cpp
  server and the default model, then the GGUF model file is ≤ 2 GB, the
  llama-server loads within 2 GB RAM, and the investment conformance test
  completes successfully.
• Given a developer runs just test-unit-all, then test-unit-fd completes
  within a per-job 1 GB workspace budget and all tests pass.
• Given a developer runs just test-system-all, then the system model test
  completes with the smaller default model.
• The ensure_hf_model.sh and run_llm_server.sh scripts default to the
  new model without requiring TK_HF_REPO_ID, TK_HF_MODEL_DIR, or
  TK_HF_MODEL_FILE environment variable overrides.

Conditional Acceptance

Financial capability and policy

• N/A - story changes test infrastructure defaults only; no policy changes.

Audit and replay

• N/A - no audit schema or replay changes; test infrastructure only.

Runtime topology and tile ownership

• N/A - no topology changes; build/test surface only.

Model, tool, adapter, or execution boundary

• Applies - changing the default system test model does not alter production
  model gateway configuration (tkmodl). Verify that the model file change
  is restricted to the test-system-tk test harness and does not leak into
  production configs or TK_HF_MODEL_* defaults outside the test scripts.

CaseOps API or UI

• N/A - no API/UI changes.

Storage role: Memory, Analytics, Ledger

• N/A - no storage role changes.

Metrics, diagnostics, and operations

• Applies - test memory footprint changes affect CI resource usage and local
  developer hardware requirements. Verify that CI lanes still complete within
  their allocated resource limits.

Security and fail-closed behavior

• N/A - test script changes do not affect fail-closed behavior or
  production security boundaries.

Child Task Issues

V2.10.S11.T1 — Architecture and planning. Survey current Firedancer test memory
usage: identify which tests exceed 1 GB, verify that the 1 GB budget does not
cause any test to fail with fd_wksp_new_anonymous out-of-pages. Select the
target smaller model for test-system-tk: evaluate Qwen2.5-1.5B vs
Qwen2.5-3B GGUF Q4_K_M models for test pass rate and inference quality.
Document the trade-off between model quality and memory budget.

V2.10.S11.T2 — Firedancer test memory reduction. Update
contrib/test/run_unit_tests.sh default JOB_MEM from the current implicit
1 GiB to an explicit JOB_MEM=1073741824 (1 GiB = 262144 normal pages), or
update the justfile test-unit-fd recipe to pass --job-mem 1073741824 in
TEST_OPTS. Update contrib/fd-build-lib.sh cov mode comment to reflect the
new budget.

V2.10.S11.T3 — System model test model swap. Update
contrib/test/ensure_hf_model.sh and contrib/test/run_llm_server.sh default
TK_HF_MODEL_FILE and TK_HF_MODEL_DIR to point to the chosen smaller model
(e.g. Qwen2.5-1.5B-Instruct-Q4_K_M.gguf). Update
contrib/test/run_system_model_tests.sh if needed. Verify the llama-server
launch command in run_llm_server.sh still works with the smaller model.

V2.10.S11.T4 — Test verification. Run just test-unit-fd and verify all
Firedancer unit tests pass with the new 1 GB budget. Run just test-system-tk and verify the investment conformance test passes with the
smaller model. Record MaxRSS or peak memory usage in the task evidence.

V2.10.S11.T5 — Documentation sweep. Update doc/execution/testing-tickoni.md
to reflect the new memory budgets. Update doc/execution/ci.md to mention the
model change. Update doc/execution/development.md prerequisites section if
hardware requirements change.

Evidence Plan

• Demo or command: just test-unit-fd (verify all tests pass, capture MaxRSS
  or peak memory), just test-system-tk (verify live investment conformance
  test passes with smaller model).
• Tests: All Firedancer unit tests pass at 1 GB/job. Live system test passes.
• Fixtures or samples: New TK_HF_MODEL_FILE default value in
  ensure_hf_model.sh.
• Audit/replay evidence: N/A - test infrastructure only.
• Blocked-flow evidence: N/A - no blocked flows.

Quality Gate

• just test-unit-fd passes all Firedancer unit tests at 1 GB/job.
• just test-system-tk passes the live investment conformance test with
  the smaller model.
• just test-unit-all passes (Tickoni + Firedancer).
• just build-all still compiles clean (no build system changes).
• Documentation and roadmap status are updated.

Notes And Open Questions

• Which specific smaller model to target? Qwen2.5-1.5B-Q4_K_M (~1 GB) is the
  safest for tight memory budgets but may produce lower-quality test output.
  Qwen2.5-3B-Q4_K_M (~2 GB) provides better quality but requires more headroom.
• Does the existing unsloth/gemma-4-E2B-it-qat-GGUF repo offer a smaller
  variant (e.g. a 7B or 1.5B model) that keeps the same ecosystem? If so,
  prefer that to reduce infrastructure friction.
• Are there Firedancer unit tests that require > 1 GB workspace that would fail
  with the reduced budget? The TEST_GIGANTIC_PAGES dictionary in
  run_unit_tests.sh already handles tests that need > 1 gigantic-page budget;
  the equivalent for normal pages may need similar per-test overrides.
• Should the 1 GB budget be a configurable default (e.g. --job-mem env var)
  so developers with more RAM can override it?