Consolidate production and v0.10 Containers into 0.4.3 candidate - #28
Merged
Merged
Conversation
release: promote v0.5 to main
GSM8K did not exist in this repo. Add it as a first-class world modeled on
Banking77, with the same honesty invariants:
- the public observation is the question only; the reference answer is
env-private and a test walks every fixture row to prove it;
- a deterministic parser reads \boxed{...}, GSM8K's native `#### N`, signed
values, decimals, fractions, comma-thousands and trailing prose, and records
a parse failure and the raw completion as separate fields;
- an unparseable or missing prediction leaves the reward null (never 0.0), and
`omit_reward` does the same;
- train/heldout split indices are persisted in the repo, and a test detects
duplicate questions and normalized-answer leakage across the two splits;
- fixtures are the default source so PR CI never downloads; real data is opt-in
via SYNTH_GSM8K_SOURCE=hf, mirroring SYNTH_BANKING77_SOURCE;
- reward is exact match on the parsed numeric answer.
Add provider id `synth_mlx_rl` so a local MLX proxy is admitted instead of
refused. One shared validator (platform/local_provider.py) is used from all
three call sites — banking77 chat, react, healthbench — on both the
chat_completions and responses families. It copies the shape of
`_validate_responses_endpoint`: userinfo/query/fragment refused, http permitted
unconditionally for loopback and host.docker.internal, every other origin
(https included) must be named in SYNTH_MLX_RL_ALLOWED_ENDPOINTS, and refusals
are terse secret-free snake_case codes the existing `_error_code` forwards.
The ReAct planner accepts `synth_mlx_rl` on chat_completions and refuses
api_family='responses' by name: it renders a chat-completions tool-call body
and has no responses-family renderer, so translating silently would evaluate a
transcript the policy was never given.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…loopback The alias resolves to the host from inside a container, so admitting it on any port handed a policy config a container-to-host probe across every service on the machine. Only 127.0.0.1/localhost/::1 stay unconditional, matching the in-repo precedent exactly; the Docker case is enabled by naming the one origin in SYNTH_MLX_RL_ALLOWED_ENDPOINTS, which is an explicit act. Restores the fail-closed shape decision D5 specified. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A completion the policy produced but that states no parseable number is a failed attempt, not an absent signal. Scoring it null was wrong in the optimistic direction: the eval layer excludes missing metrics from the denominator, so a model that rambles instead of answering would have its worst trials deleted and its accuracy computed only over the subset where it happened to emit a number. That is exactly the failure mode of a small base model, which is the first thing this lane measures. Genuinely absent signals still stay null: no completion at all, omit_reward, or an env row whose own reference will not parse. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ReAct planner previously refused api_family='responses' with a named gap. It now renders a Responses body and reads a function call back, normalizing into the same chat-shaped dict the parse/retry/usage/compaction paths already consume -- the same approach _tinker_sample already uses. Three differences from chat, each of which silently breaks a rollout if missed: the tool is flat rather than nested under 'function', the cap is max_output_tokens, and usage arrives as input_tokens/output_tokens. The last matters most here: compaction triggers on prompt_tokens, so an unmapped usage block reads as a context that never grows and the transcript is never compacted. _summarize speaks the family too; a chat body posted to a /responses route is a 4xx, and a failed summary silently drops the turns it was compacting. D9 now holds at all three call sites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The proxy owns the authoritative token ids and rollout logprobs; this container owns the reward. Nothing joined them, so an on-policy trace was unusable however complete each half was on its own. The runtime now emits a token_capture event carrying proxy_request_ids, policy_snapshot_id and the tokenizer/template digests, and token_capture is declared on the target's event_kinds. Deliberately a reference, never the tokens: TokenCaptureV5 already refuses a sequence that is both inline and artifact-backed, and a container relaying a training record it does not own is what the proxy design exists to avoid. No event is emitted when there is no proxy record (gold, forced, hosted provider) -- claiming a record that never existed would be worse than none. 447 passed, 9 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…split
Two defects that together made the task unmeasurable rather than hard. Both
found by running the real Qwen3.5-0.8B against it, not by reading the code.
1. The 77-label vocabulary was never in the observation. CLASSIFY_SYSTEM told
the policy to "return exactly one label from the allowed label list" and no
list was ever supplied, so it had to emit one exact string out of 77 it had
never seen. Measured: 0/40, with every prediction a plausible intent name
(`locate_card`, `pending`, `delivery_timing`) that simply is not in the
vocabulary. That measures the prompt, not the model. Plan section 9 already
required "constrain output to the pinned label vocabulary".
The vocabulary is not gold: it is identical for every item and therefore
carries no per-item signal. The leakage test now asserts that stronger
property -- two items with different answers get an identical label list --
instead of the weaker "the gold string does not appear", which the action
space necessarily violates.
2. PolyAI/banking77 ships label-sorted, so seeds 0..N drew N consecutive items
of the SAME class: seeds 0-3 of the test split were all `card_arrival`. Rows
are now shuffled once with a fixed seed before seeds index into them. The
seed constant is fixed forever; changing it renumbers every seed and
silently invalidates any previously reported number.
Effect on the real held-out baseline, Qwen3.5-0.8B base, temp=0, n=40:
before 0/40 = 0.000
after 18/40 = 0.450 (random over 77 classes is 0.013)
448 passed, 10 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ders
`reasoning: {effort}` is an OpenRouter extension, not part of the OpenAI chat
schema. It was sent on every request regardless of provider, so a strict server
rejected the whole call with a 422 -- which surfaces as `policy_error` and
reads as the model failing rather than the request being malformed.
Found by driving the planner against the real local synth_mlx_rl service, whose
request models forbid extra fields. The strictness is what caught it.
Verified multi-turn on Qwen3.5-0.8B, three turns, both api families:
chat_completions prompt_tokens 820 -> 1840 -> 3060, legal actions
responses prompt_tokens 820 -> 1840 -> 3060, legal actions
The token counts being identical across families is the D9 one-renderer
invariant showing up empirically: both surfaces really do render one transcript.
449 passed, 10 skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mentation # Conflicts: # src/synth_containers/platform/runtimes/healthbench.py
release: land v0.6 local MLX runtime support
…ampler Two changes, found by driving this container from a local on-policy trainer. **The training sampler path hardcoded `temperature: 0.0`.** The typed training boundary validates a temperature and plumbs it into the policy config, and `_sample_remote_checkpoint` then ignored it and asked for greedy decoding. Every sample in a rollout group is therefore identical, so the group has no reward variance, so group-relative advantages are undefined and the step is filtered. A group-based RL algorithm cannot learn anything from this container as it stands -- and a bounded CISPO canary against Banking77 reported exactly that shape: eight rollouts, four groups, zero reward variance, no optimizer step, no checkpoint. That result is currently the stated reason CISPO is `not_validated` in the hosted model catalog. This does not prove the canary would have passed, but it does mean the run could not have produced a signal whatever the model did. Greedy remains the default for callers that ask for nothing. **A wholly local training run could not reach its own sampler.** The rollout boundary requires an HTTPS sampler URL, which is right for the hosted lane -- the cloud reaches the sampler across the public internet. When both ends are on one machine there is no network to protect and the requirement blocks the lane outright. `SamplerEndpoint.validate` already had `allow_loopback_http` and a real loopback-host check; nothing passed it. Now `SYNTH_CONTAINERS_ALLOW_LOOPBACK_SAMPLER` does, off by default, and the host check still means a remote plaintext endpoint is refused however the flag is set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`public_observation` computes the 77-label vocabulary and advertises it, and then every harness built its user turn with `user_prompt(text)` -- no labels. The policy was asked to emit one of 77 exact strings, scored by exact match, while never being shown the list. That is not a fair task and it is not a measurable one: a model that understands the query perfectly still has to guess the label's exact spelling and underscores. Asked about card fraud, Qwen3.5-0.8B answers `fraud`, which is reasonable and scores zero. Every rollout in a group scores zero, the group has no reward variance, and the on-policy lane filters it -- so the environment could not produce a learning signal at all. All four prompt sites now pass the vocabulary the observation already carries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The grader took `text.strip().splitlines()[0]`, which is only correct for a
model that answers immediately. gpt-oss-20b -- the model this task is graded on
in the hosted catalog -- is a reasoning model. Sampled on a real Banking77
prompt it returns:
<|channel|>analysis<|message|>We need to classify into exactly one intent.
Query: "Why can't I add this beneficiary?"
Check labels: There's "beneficiary_not_allowed" maybe used for ...
It reasons its way to the correct label and the grader reads line one, so the
rollout scores 0.0. Forty consecutive hosted rollouts scored 0.0 this way, every
group was filtered for zero reward variance, and no optimizer step was taken --
which is the same shape as the bounded canary whose result is the stated reason
CISPO sits at `not_validated` in the hosted catalog.
So there were two independent guarantees of that outcome: the hardcoded
`temperature: 0.0` fixed earlier, and this. Either alone makes reward variance
impossible, and neither is visible in any metric the run reports.
`extract_answer` prefers the harmony `final` channel and otherwise takes the
last non-empty line, which is where a model that thinks out loud puts its
answer. All four extraction sites use it. A model that never reaches a final
answer within `max_tokens` still scores zero, correctly -- but note that
budgeting 24 tokens for a reasoning model guarantees that, so callers must size
`max_tokens` for the reasoning too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extract_answer looked for <|channel|>final<|message|>, which the hosted checkpoint sampler removes: sample_hosted_tinker_checkpoint.py decodes with skip_special_tokens=True. What arrives is one blob whose channel names survive as ordinary text -- '...assistantfinalbeneficiary_not_allowed' -- so the last line is the entire chain of thought and a correct answer scores zero. Measured side by side on one prompt: decoding with the markers kept extracts 'beneficiary_not_allowed'; the stripped decode the run actually receives extracted the whole reasoning trace. That is why the model scores 5/5 when sampled directly and 0/32 through the run. Now handled both ways, so the grader works whether or not the caller preserved the control tokens. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sampler Honour sampler temperature and allow loopback; banking77 grading fixes
A group-relative algorithm sampling at temperature 0 cannot produce a learning signal. Point Craftax at SamplerEndpoint, attach the per-call token receipt, and admit loopback HealthBench the same way banking77 does. Co-authored-by: Cursor <cursoragent@cursor.com>
…sampler Craftax hosted sampler: honour temperature, SamplerEndpoint, token receipt
…rial; loopback receipt The world pins openai/gsm8k main @ 740312add88f781978c0658806c59bc2815b9866 in code (HF_REVISION) with a digest per split (train 7473 rows sha256:dca44988…, test 1319 rows sha256:32c548f0…) and refuses rows that do not reproduce them. Which rows back the world is a declared profile (fixture / hf / snapshot) set in code; SYNTH_GSM8K_SOURCE=hf survives only as a test-time opt-in and can no longer be the authority for anything reportable. Seeds index a recorded permutation (SHUFFLE_SEED) of each pinned split. dataset_manifest() is exposed on /metadata and hashed into the capability digest. Every action now carries parse_mode (exact / trailing_number / unparsed) and format_compliant, so a 0% parse-failure rate cannot be read as format compliance: only `exact` trials used a marked answer. The gsm8k runtime learns the typed training boundary's sampler path (/training/rollouts -> /v1/training/sample), mirroring Banking77, with a 4096-token ceiling instead of Banking77's 32. docs/receipts/2026-08-20/gsm8k-loopback/: one mechanism receipt against a real v0.7 synth-mlx-rl (Qwen3.5-0.8B, offline) through both paths, with the token_capture (proxy_request_ids + policy_snapshot_id) join resolved. Tests: tests/test_gsm8k_pin.py (18) incl. an offline verification of the real pinned revision when it is in the HF cache. Pre-existing failure untouched: test_banking77_platform.py::test_remote_checkpoint_sampler_is_loopback_only_and_secret_free fails identically at the base commit. Base: origin/v0.7 9ed2597 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gsm8k: pin openai/gsm8k by revision + split digests, parse mode per trial, loopback receipt (L2)
…marker-fix fix(banking77): strip collapsed final channel marker
…sampler v0.7: land the Banking77 sampler fixes on the release branch
release(v0.7): promote v0.7 to main
…çade CompatPlatform and create_compat_app no longer fall back to tempfile.mkdtemp; the durable root is named by the caller (tests pass tmp_path, examples and scripts create their own). The Harbor nested child platform now lives under the parent's root keyed by the rollout digest. Lock: tests/test_storage_root_required.py (factory refuses a missing root; no mkdtemp/TemporaryDirectory in state.py, app.py, extensions/dock.py). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PivnC4AEhuoEMh7vBsaQy5
Add tests/code_quality/test_file_size_cap.py enforcing a 2,000-line cap
on Python files in the repo (D-X-2). Current offenders are recorded on
an explicit allowlist with their line count at time of writing:
- src/synth_containers/tracing/capture/finalizer.py (2085)
- src/synth_containers/tracing/capture/supervisor.py (2331)
- src/synth_containers/tracing/validation/validator.py (4434)
- tests/test_trace_v5_capture_security_regressions.py (2549)
Any non-allowlisted file over the cap fails the test; an allowlisted
file that drops back under the cap also fails, telling the author to
remove it from the list, so the list can only ever shrink.
Collected under default pytest invocation (no explicit path needed):
uv run --with pytest pytest --collect-only -q
Verify: uv run pytest tests/code_quality/test_file_size_cap.py -q
(3 passed)
Also adds receipts/L-C-2026-08-21.md with the Phase 2/4 line-number
refresh for P2-4, P2-5, P4-1, P4-2 (read-only inventory, no code
changes for those items).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017BaixTeAyFEH4deiDNpUEg
containers: P0-9 file-size cap and P2-4 explicit storage_root
Route simulate exceptions through the same terminalization as runtime _fail (status events, seal, completed manifest, drop lease). Leases now carry policy_ref/task_instance_id/seed/digests and recovery fails closed without them. PinStatus.transition is the status writer; frozen AdmissionReceipt makes replay a digest compare. Split pin/manifest types out of state.py to stay under the P0-9 2,000-line cap. Lock: tests/test_crash_mid_rollout.py (uvicorn SIGKILL mid-simulate recovers crashed with the real task_instance_id and refuses the unsealed log). Co-authored-by: Cursor <cursoragent@cursor.com>
…-digest-v2 Publish the pinned NanoHorizon Craftax runtime catalog
Port 6275e41, a559169, 09b1bbb, f3263aa, 8010e54 and b8e6490 onto production history. Preserve main task limits, terminal projection, SSE control deduplication, journal ack/hash/retention, and both reward authorities. Remove duplicated status enums from the merge. 55 cross-contract and 62 annotation tests pass; 19 journal tests pass including filtered-page ack/retention coverage. Remaining v0.10 groups and release version are not yet integrated.
…ta contracts Port merge delta 5472530 against its first parent. Preserve main structured event-page OpenAPI, metadata composition, closed timestamp and hash/ack methods, immutable policy checks and gold world cleanup. Add annotation listeners and bounds without duplicating validation. 72 focused live annotation, HTTP, bounds, journal and metadata tests pass.
…n bumps Port runtime and tests from 232c213; retain working-tree version until the reconciled release is cut as a new immutable package.
Port 9da5d19 while retaining bounded NanoHorizon recovery, capability budgets, immutable policy revisions, explicit provenance and journal acknowledgements. Scan non-durable events before publication as well. Adapt older fixtures to current required limits and terminal reward contracts.
Preserve production APIs alongside v0.10 implementation groups. Full suite: 1028 passed, 10 skipped; Ruff clean; 172 existing type diagnostics with no new signatures. Split handshake facts and platform records to honor the existing size cap, preserving old imports. Keep exceptional inspection-server cleanup and public no-provider smoke coverage. Candidate branch CI cannot publish; 0.4.2 remains immutable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Production-based integration from e132b9e, preserving main API, metadata, immutable policy revisions, terminal projection and journal-v2 contracts. Curated v0.10 groups add annotation/reward streams, mini-SWE, runtime tracing/checkpoints, CISPO, safe recovery, tunnel WS/SSE and explicit event artifact links. Dev is behind main; the PR preserves production ancestry.
Evidence at 81dbe89
Release boundaries
Published 0.4.2 is immutable; this is unpublished 0.4.3. Tag only after hosted CI/review and dev -> staging -> main promotion, then verify protected PyPI publication and a public-index install. Re-pin Optimizers/Workshop with coordinated package/source/lock evidence. Live GEPA E2E remains user-deferred, not verified; no paid calls were made. TBLite remains eval-only. Default-branch Dependabot findings still need disposition before final release.