v0.10 Optimizers 0.2.22 consolidation review (not merge-ready) - #54
Closed
JoshuaPurtell wants to merge 169 commits into
Closed
JoshuaPurtell wants to merge 169 commits into
JoshuaPurtell wants to merge 169 commits into
Conversation
…se-workflow-main-20260814 Add trusted PyPI release workflow
release: promote optimizers 0.2.13
release: promote optimizers 0.2.14
…ncile release: reconcile and promote v0.5 to main
Baseline at this commit: 51 passed. Records the audited blockers (ModelRoute https-only, SftConfig backend enum, missing mlx-lora policy kind) and the containers pin coupling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four narrow changes let `eval` measure a locally-served policy without giving a candidate any new say in where inference goes. O1 `ModelRoute` accepts a local route. `https://` stays unrestricted; `http://` is permitted only for a host that cannot be anywhere but this machine (127.0.0.1, localhost, ::1, host.docker.internal). Parsed with `urlsplit`, mirroring `_validate_remote_checkpoint_endpoint` in the containers Banking77 runtime but without its environment-variable allowlist — the recipe catalog is already the allowlist, and a second one read from the environment would be an escape hatch around it. Userinfo, query strings, and fragments are refused, and the path must name one of the two first-class API families. Private ranges stay refused: "it is on my LAN" is not the claim "it cannot leave this machine". O2 needed no schema change. 0.0 was already a legal rate, so a zero-rate route with `price_source = "local-compute"` and a local bearer token in `secret` is expressible today; the tests pin that shape, and that `budget.max_llm_calls` remains the ceiling that actually bounds a run once `max_usd` is vacuous. O3 adds the `mlx-lora.v1` policy kind. The adapter bytes live in the candidate, so `artifact_digest` is the digest of the adapter and base-vs-checkpoint-vs-final is one content-addressed set scored on shared seeds. `policy.json` declares `adapter` explicitly rather than letting it be inferred from which files are present, because a checkpoint whose adapter failed to copy would otherwise be scored as its own baseline and report a lift of zero. O4 registers the candidate with a snapshot registrar before each trial and writes the returned id into `trial.json`. The registrar is injected and has no default: a run with adapter candidates and no registrar fails before it starts a container, and a run without one never opens a socket. An id that drifts inside one run is refused — that means the service handed back a mutable name. O5 adds `eval.mlx.local-policy.smoke.v1` for GSM8K. `network = "bridge"` is mandatory (a `none` network cannot reach a host proxy) and `required_artifacts` is `["trace"]`. No `image_digest` is set, so the recipe is honestly unavailable until the target is published and pinned rather than carrying a fake one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pinned rev's banking77.py (346 lines) has no _sample_responses and no _validate_responses_endpoint; containers main (a2a316b) has both. D9 does not hold on this side until the pin is bumped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous pin predated the Banking77 responses lane entirely, so D9's responses-first-class claim could not hold on this side. No API drift: 88 passed before and after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Was pinned to a2a316b2, which predates both the hosted rollout contract and the MLX lane — `synth_containers.platform.local_provider` does not exist there, so the pin disagreed with the local-MLX eval this branch ships. Repinned to 5453731dabc078fc4aae700015f7ecd2ae95a969, the immutable SHA of containers v0.6/implementation on origin, and relocked. Verified by a --no-cache install into a fresh venv resolving from origin alone: the module imports and host.docker.internal is still outside the blanket-loopback set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
release: land v0.6 local MLX optimizer lane
release: synth-optimizers 0.2.15
An optimizer run an experiment dispatched has to be findable from the
experiment side, and the arm a rollout belongs to has to be readable from
the rollout side. One small record carried unchanged through the run
request, the resolved config, the manifests, and the run registry makes
both directions work without any of those growing a private notion of
what an arm is.
Two deliberate omissions in the envelope.
There is no run id. The service still mints its own and reports it back;
a caller supplying one would be asserting authority over a namespace it
does not own, and resume would then have two candidate truths about which
run a trial is. deny_unknown_fields refuses one that tries.
There is no digest. The producer computes one from its own canonical
encoding and compares it against the envelope it gets back. Recomputing
it here would mean two languages agreeing byte for byte on JSON
canonicalisation forever, for a field nothing on this side reads.
Optional fields are skipped rather than nulled, because the envelope has
to survive being written into an executor's own config format and read
back byte-identical, and TOML has no null.
Surfaces:
POST /runs correlation, validated before the contract handshake,
while no run record exists yet
RunConfig carried, never read
GET /runs/{id} echoed, next to the run id the service minted
result_manifest success and failure alike -- a failed trial still has
to join to its arm
run_registry so "every run belonging to trial X" needs no second
index
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dy exist
An ablation is only worth anything when the only thing that differed is
the thing you meant to change. This is the thin layer that makes that
guarantee cheap: you declare what varies, and it proves nothing else did.
It does three things -- assign, adapt, reduce -- and refuses the fourth.
eval, the matrix runner, and GEPA already own execution, queueing, and
evidence; the moment this layer owns them too there are two stories about
what ran.
A spec says what varies and what must not. It never says how to run
anything: the image, the seed schedule, the metrics, and the ceilings all
come from the executor's own trusted configuration, and the spec may only
select among what is already declared there. That asymmetry is what makes
an ablation cheap to write and still impossible to accidentally turn into
an unfair comparison.
Three rules do most of the reducer's work:
a failed trial is missing evidence, never a zero;
missingness is harmless only when symmetric, so it is measured per arm
and the difference gates the claim;
the order trials actually ran in is evidence, and nominal
counterbalancing is not.
headline_claim_allowed needs the declared sample, the confidence
interval, completion, missingness, image identity, isolation, and start
order all to pass. An interval containing zero closes the gate rather
than annotating it: a null result is a real finding the report states
plainly, but it cannot carry a directional headline.
Two adapters ship because their executors do.
eval.runtime drives one eval run per cell of the matrix, which buys an
exact dispatch order to counterbalance and one separately-resumable
sealed receipt per trial -- neither of which a single batched run gives.
Every override it sends narrows the trusted recipe.
gepa.cli renders one config per trial through the ordinary GepaRun path.
Its factor catalog is curated rather than reflected: GEPA's TOML sections
are extra="ignore", so a schema walk would offer knobs the engine
silently drops. max_generations, minibatch_size, and the budget caps are
excluded on purpose -- varying how much work an arm does voids any
wall-clock or cost comparison between arms.
evals.matrix and gepa.service register from their own repositories via
register_adapter, so this layer takes no dependency on every executor it
can drive.
Also cuts 0.2.15, so Workshop can pin an Eval runtime carrying the
2ed30aa admission-readiness hardening.
Verified against the real eval-fixture-target container: A/A over three
blocks with a delta of exactly zero (no cache, container, or seed leak),
then a two-arm four-block comparison walked end to end from outcome row
through sealed correlation, narrowed seeds, the trial manifest the
container read, job_result, and the digested trace artifact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nder `experiment plan` printed "image UNPINNED" for every executor, because it looked for an image digest that only the eval runtime records. gepa.cli pins a base config digest, a taskset digest, and a task-pools digest instead. Show what the adapter actually recorded rather than one executor's vocabulary applied to all of them. Also pins, in config.rs, the exact TOML that gepa_cli renders into a trial config. RunConfig is deny_unknown_fields, so that parse is where a correlation envelope would be rejected -- before a container starts and long before anyone notices a manifest that cannot be joined to an arm. TOML has no null, so the test also asserts the optionals arrive absent rather than empty; a `candidate_id = null` would not have parsed at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… names Checked the adapter against a real sealed manifest (banking77_gepa_async_t50_mb20_h100_735a9c29) rather than the fake one the tests build, and found two things the fake had hidden. GEPA's usage block reports `rollout_calls`, not `rollouts`. A spec asking for `rollouts` as a secondary metric produced an empty comparison with no error anywhere -- which is the same silent-reduction failure the layer exists to prevent, one level up. Lifting every numeric scalar that `best_candidate` and `usage` actually carry removes the whole class. A successful run's manifest also has no `status` key at all; it is the serialised GepaRunResult. Only failure manifests carry status. The seal already fell through to the right branch, but by luck, so the real shape is now pinned in a test. Resource metrics (tokens, calls) join cost and wall time as improving downward. For a reasoning-effort ablation "the harder-thinking arm spent more" is the expected finding, not a regression -- the direction only fixes the sign convention. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sealed failure was skipped on resume forever, so one crashed container five hours into a twenty-trial matrix permanently poisoned that block and, under missing_policy = "fail", blocked the claim outright. Restarting the whole experiment was the only remedy. `resume --retry-rig-failures` re-dispatches trials whose failure was the rig's. Only `rig` and `infra` qualify: a `policy` failure is the thing under test, and a `budget` or `timeout` failure may itself *be* the arm difference, so retrying either would be selecting for the result you wanted. Nothing is hidden. A retry appends a row carrying `attempt = N+1` and the digest of the row it supersedes; the superseded rows stay in the log, `retried_trials` lands in the report totals, and the claim verdict names them in a note. A rig that needed three attempts is a fact about the comparison, not housekeeping. Two rows for the same attempt are still a contradiction rather than a retry. Each retry runs under its own executor identity (`…_r1`). Found by test: the first implementation re-dispatched correctly and still completed zero trials, because the eval runner treats an existing job_result.json as terminal and replayed the very failure the retry existed to clear. Every executor here resumes its own work from a sealed record, so the attempt number has to reach the adapter and change the run identity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Serial dispatch made the planned order the executed order, and made a twenty-trial matrix a sixty-five minute wait. `[execution] max_parallel_trials` fans out while still submitting in planned order. For a *paired* design this is arguably better than serial rather than a compromise: both arms of a block now meet the same machine in the same minute, which removes temporal drift instead of adding it. What it does add is contention, which is why the reducer keeps computing fairness from recorded dispatch and start times rather than trusting the setting -- the run this landed for observed a start-order bias of 0.021 against a declared limit of 0.25. The knob is declared in the spec and carried in the plan, so a report can compare intent against what happened. Serial stays the default. A port per trial is the load-bearing half. A base config names one container socket, so concurrent trials would have fought over it -- and a collision would not have looked like an error, it would have looked like one arm scoring the other arm's rollouts. Ports derive from the trial id, are asserted distinct across the matrix, and `container.url` joins the reserved paths a spec cannot make a treatment. The launch command's port literal is rewritten too, since moving only the URL leaves the trial talking to nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every charge in every v0.10 acceptance ledger settled `conservative`, and all 1,624 recorded Tinker receipts carry full token accounting with no cost at all. So the counted aggregate is a sum of declared reservation prices, not of money the provider reported — and nothing in the suite said so. The nearest existing coverage settles unknown cost at the `ExperimentBudget` level and exercises `TrainingBudget` only with zero session/save/restore fees. That misses the case that dominates real spend: under C1's contract those three lanes are a flat $0.25 the provider is never asked about, and 55 of them are 98% of the counted aggregate. These tests pin the distinction — that a real Tinker payload parses to `cost_missing`, that absent cost records `conservative` at the reservation, that the flat lanes charge their declared price whatever the work, and that reported cost still records `usage_counted`. The last is the branch Tinker never takes today; it has to stay reachable and distinct, or nothing separates a measured charge from a reserved one. Payload shapes are taken from real receipts in the retained C1 ledger; the transports and budget under test are the production ones. Each test owns a tmp_path store and makes no provider call. Verified against the installed 0.2.21 candidate, whose bytes match this pinned source for every file under test: 7 passed, and 39 passed with the neighbouring durability and budget suites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YMzDUNws4mWSWkyLGuE5cd
98% of the v0.10 counted training aggregate is flat per-operation reserves for session, save and restore — 55 charges at a declared $0.25 against $0.234719 of token-metered work. The accounting failure that would cost real money is therefore not a mispriced token: it is a save reserved twice across a crash, or a cap that comes back empty after a restart. The existing durability suite drives restart and fencing with those three lanes priced at zero, so neither fault can appear in it. These six tests drive the same faults under C1's real pricing contract: a save reserved before a crash cannot be reserved again on reopen; a spent cap is not handed back; a resumed run cannot come back with a larger cap; an unsettled flat operation does not free a slot; a settlement survives reopen and cannot be restated in either direction; and two jobs sharing one experiment cannot oversubscribe it across a restart. No provider call, no credential, one tmp_path store per test. 6 passed against the installed 0.2.21 candidate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YMzDUNws4mWSWkyLGuE5cd
…ompiling `RELEASE.md` declares cargo fmt, check, clippy, ruff and ty as the validation set. clippy failed with 14 errors and ruff with 2 at the release pin. `cargo test` was not in that set, and had rotted: `identities.rs` used `LeverBundle` without importing it, so the crate's tests had stopped compiling entirely. Adding the import brings 137 tests back. The gate is now listed so it cannot rot again unnoticed. Two clippy findings were real. `DEFAULT_LEAKAGE_MIN_SPAN_CHARS` was declared in `synth_gepa::leakage` and never read, while `synth_optimizer_platform::config` carried its own literal `32` for the same default — two copies, one of them dead, exactly how the two drift. One authority now lives in `levers.rs`, which both read. And `ServiceOwnershipGuard`'s lock-path accessors are used only by tests, so they are scoped to the test build rather than deleted. The rest are legitimate: eleven constructors and snapshot builders take eight or more distinct domain values, and a parameter struct would only move the arity into a literal at every call site. That is now one workspace lint table with one reason, rather than eleven attributes scattered through source files that a size ratchet allows only to shrink. `cargo fmt --check` and that ratchet disagree, and this commit does not resolve it. Formatting the four allowlisted files adds 12, 3, 6 and 6 lines, which turns `file_size_cap` red; those four are left unformatted so the test holds, and every other file is formatted. Whether to split them or re-cut the ceilings is a review decision, and raising a ceiling quietly is the one thing the test exists to prevent. cargo check, clippy -D warnings, ruff, py_compile and git diff --check are clean; 137 Rust tests pass. ty still reports 226 diagnostics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YMzDUNws4mWSWkyLGuE5cd
The 0.2.21 bytes are the distribution Workshop v0.10 embeds, pinned by
source revision, so the gate fixes ship as the next dev build rather
than changing what that version means.
Re-pins the package family. `synth-optimizers` pins `synth-containers`
and `synth-harbor-tblite` exactly, and the adapter pins Containers
exactly in turn, so a Containers bump forces all of them to move
together — resolution fails outright otherwise. Both vendored wheels are
replaced and the stale dev20260903 and dev20260908 copies removed.
The publish workflow ran maturin and nothing else: no tests, no lint. It
now runs the Rust suite, clippy, ruff and `git diff --check` before
either build job, so the suite that had stopped compiling cannot rot
unnoticed again.
137 Rust tests pass, clippy and ruff clean. Built wheel:
synth_optimizers-0.2.22.dev20260909-cp311-abi3-macosx_11_0_arm64.whl
sha256 2a3f2316cbe6cf42ae8ea7f24d2d136ccc2a829b98099126af170b9952d95d4d
Requires-Dist: synth-containers==0.4.2.dev20260909
Built locally and not published. This version matches the dev channel
pattern only; an official cut needs Containers stable first, because a
stable release must not pin a dev build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMzDUNws4mWSWkyLGuE5cd
Contributor
Author
|
Superseded by #55, the curated integration branch. The candidate source remains preserved on codex/v010-stable-packages; no branch or worktree is being deleted. #55 retains production 0.2.16 history and ports v0.10 as explicit implementation groups. Runtime/tests/dependency bytes match the validated candidate; only integration CI configuration, restored changelog, and inherited documentation whitespace differ. Closing this review anchor without merging the old stack. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Consolidate the v0.10 Optimizers candidate into the integration line, then promote through staging and main. This draft is a review anchor, not authorization to bulk-merge the inherited stack.
Current evidence
Required before integration/promotion
Scope boundaries
TBLite remains eval-only. Uncommitted primary-worktree MAPO/research work is not included. The broader duplicate cache change in the old acceptance worktree must not be blindly applied over c21d6fe. Preserve dirty worktrees; no stash/reset/cleanup.