Integrate v0.10 Optimizers 0.2.22 with preserved production history - #55
Merged
Merged
Conversation
…se-workflow-main-20260814 Add trusted PyPI release workflow
release: promote optimizers 0.2.13
release: promote optimizers 0.2.14
…ncile release: reconcile and promote v0.5 to main
Baseline at this commit: 51 passed. Records the audited blockers (ModelRoute https-only, SftConfig backend enum, missing mlx-lora policy kind) and the containers pin coupling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four narrow changes let `eval` measure a locally-served policy without giving a candidate any new say in where inference goes. O1 `ModelRoute` accepts a local route. `https://` stays unrestricted; `http://` is permitted only for a host that cannot be anywhere but this machine (127.0.0.1, localhost, ::1, host.docker.internal). Parsed with `urlsplit`, mirroring `_validate_remote_checkpoint_endpoint` in the containers Banking77 runtime but without its environment-variable allowlist — the recipe catalog is already the allowlist, and a second one read from the environment would be an escape hatch around it. Userinfo, query strings, and fragments are refused, and the path must name one of the two first-class API families. Private ranges stay refused: "it is on my LAN" is not the claim "it cannot leave this machine". O2 needed no schema change. 0.0 was already a legal rate, so a zero-rate route with `price_source = "local-compute"` and a local bearer token in `secret` is expressible today; the tests pin that shape, and that `budget.max_llm_calls` remains the ceiling that actually bounds a run once `max_usd` is vacuous. O3 adds the `mlx-lora.v1` policy kind. The adapter bytes live in the candidate, so `artifact_digest` is the digest of the adapter and base-vs-checkpoint-vs-final is one content-addressed set scored on shared seeds. `policy.json` declares `adapter` explicitly rather than letting it be inferred from which files are present, because a checkpoint whose adapter failed to copy would otherwise be scored as its own baseline and report a lift of zero. O4 registers the candidate with a snapshot registrar before each trial and writes the returned id into `trial.json`. The registrar is injected and has no default: a run with adapter candidates and no registrar fails before it starts a container, and a run without one never opens a socket. An id that drifts inside one run is refused — that means the service handed back a mutable name. O5 adds `eval.mlx.local-policy.smoke.v1` for GSM8K. `network = "bridge"` is mandatory (a `none` network cannot reach a host proxy) and `required_artifacts` is `["trace"]`. No `image_digest` is set, so the recipe is honestly unavailable until the target is published and pinned rather than carrying a fake one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pinned rev's banking77.py (346 lines) has no _sample_responses and no _validate_responses_endpoint; containers main (a2a316b) has both. D9 does not hold on this side until the pin is bumped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous pin predated the Banking77 responses lane entirely, so D9's responses-first-class claim could not hold on this side. No API drift: 88 passed before and after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Was pinned to a2a316b2, which predates both the hosted rollout contract and the MLX lane — `synth_containers.platform.local_provider` does not exist there, so the pin disagreed with the local-MLX eval this branch ships. Repinned to 5453731dabc078fc4aae700015f7ecd2ae95a969, the immutable SHA of containers v0.6/implementation on origin, and relocked. Verified by a --no-cache install into a fresh venv resolving from origin alone: the module imports and host.docker.internal is still outside the blanket-loopback set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
release: land v0.6 local MLX optimizer lane
release: synth-optimizers 0.2.15
An optimizer run an experiment dispatched has to be findable from the
experiment side, and the arm a rollout belongs to has to be readable from
the rollout side. One small record carried unchanged through the run
request, the resolved config, the manifests, and the run registry makes
both directions work without any of those growing a private notion of
what an arm is.
Two deliberate omissions in the envelope.
There is no run id. The service still mints its own and reports it back;
a caller supplying one would be asserting authority over a namespace it
does not own, and resume would then have two candidate truths about which
run a trial is. deny_unknown_fields refuses one that tries.
There is no digest. The producer computes one from its own canonical
encoding and compares it against the envelope it gets back. Recomputing
it here would mean two languages agreeing byte for byte on JSON
canonicalisation forever, for a field nothing on this side reads.
Optional fields are skipped rather than nulled, because the envelope has
to survive being written into an executor's own config format and read
back byte-identical, and TOML has no null.
Surfaces:
POST /runs correlation, validated before the contract handshake,
while no run record exists yet
RunConfig carried, never read
GET /runs/{id} echoed, next to the run id the service minted
result_manifest success and failure alike -- a failed trial still has
to join to its arm
run_registry so "every run belonging to trial X" needs no second
index
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…dy exist
An ablation is only worth anything when the only thing that differed is
the thing you meant to change. This is the thin layer that makes that
guarantee cheap: you declare what varies, and it proves nothing else did.
It does three things -- assign, adapt, reduce -- and refuses the fourth.
eval, the matrix runner, and GEPA already own execution, queueing, and
evidence; the moment this layer owns them too there are two stories about
what ran.
A spec says what varies and what must not. It never says how to run
anything: the image, the seed schedule, the metrics, and the ceilings all
come from the executor's own trusted configuration, and the spec may only
select among what is already declared there. That asymmetry is what makes
an ablation cheap to write and still impossible to accidentally turn into
an unfair comparison.
Three rules do most of the reducer's work:
a failed trial is missing evidence, never a zero;
missingness is harmless only when symmetric, so it is measured per arm
and the difference gates the claim;
the order trials actually ran in is evidence, and nominal
counterbalancing is not.
headline_claim_allowed needs the declared sample, the confidence
interval, completion, missingness, image identity, isolation, and start
order all to pass. An interval containing zero closes the gate rather
than annotating it: a null result is a real finding the report states
plainly, but it cannot carry a directional headline.
Two adapters ship because their executors do.
eval.runtime drives one eval run per cell of the matrix, which buys an
exact dispatch order to counterbalance and one separately-resumable
sealed receipt per trial -- neither of which a single batched run gives.
Every override it sends narrows the trusted recipe.
gepa.cli renders one config per trial through the ordinary GepaRun path.
Its factor catalog is curated rather than reflected: GEPA's TOML sections
are extra="ignore", so a schema walk would offer knobs the engine
silently drops. max_generations, minibatch_size, and the budget caps are
excluded on purpose -- varying how much work an arm does voids any
wall-clock or cost comparison between arms.
evals.matrix and gepa.service register from their own repositories via
register_adapter, so this layer takes no dependency on every executor it
can drive.
Also cuts 0.2.15, so Workshop can pin an Eval runtime carrying the
2ed30aa admission-readiness hardening.
Verified against the real eval-fixture-target container: A/A over three
blocks with a delta of exactly zero (no cache, container, or seed leak),
then a two-arm four-block comparison walked end to end from outcome row
through sealed correlation, narrowed seeds, the trial manifest the
container read, job_result, and the digested trace artifact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…nder `experiment plan` printed "image UNPINNED" for every executor, because it looked for an image digest that only the eval runtime records. gepa.cli pins a base config digest, a taskset digest, and a task-pools digest instead. Show what the adapter actually recorded rather than one executor's vocabulary applied to all of them. Also pins, in config.rs, the exact TOML that gepa_cli renders into a trial config. RunConfig is deny_unknown_fields, so that parse is where a correlation envelope would be rejected -- before a container starts and long before anyone notices a manifest that cannot be joined to an arm. TOML has no null, so the test also asserts the optionals arrive absent rather than empty; a `candidate_id = null` would not have parsed at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… names Checked the adapter against a real sealed manifest (banking77_gepa_async_t50_mb20_h100_735a9c29) rather than the fake one the tests build, and found two things the fake had hidden. GEPA's usage block reports `rollout_calls`, not `rollouts`. A spec asking for `rollouts` as a secondary metric produced an empty comparison with no error anywhere -- which is the same silent-reduction failure the layer exists to prevent, one level up. Lifting every numeric scalar that `best_candidate` and `usage` actually carry removes the whole class. A successful run's manifest also has no `status` key at all; it is the serialised GepaRunResult. Only failure manifests carry status. The seal already fell through to the right branch, but by luck, so the real shape is now pinned in a test. Resource metrics (tokens, calls) join cost and wall time as improving downward. For a reasoning-effort ablation "the harder-thinking arm spent more" is the expected finding, not a regression -- the direction only fixes the sign convention. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sealed failure was skipped on resume forever, so one crashed container five hours into a twenty-trial matrix permanently poisoned that block and, under missing_policy = "fail", blocked the claim outright. Restarting the whole experiment was the only remedy. `resume --retry-rig-failures` re-dispatches trials whose failure was the rig's. Only `rig` and `infra` qualify: a `policy` failure is the thing under test, and a `budget` or `timeout` failure may itself *be* the arm difference, so retrying either would be selecting for the result you wanted. Nothing is hidden. A retry appends a row carrying `attempt = N+1` and the digest of the row it supersedes; the superseded rows stay in the log, `retried_trials` lands in the report totals, and the claim verdict names them in a note. A rig that needed three attempts is a fact about the comparison, not housekeeping. Two rows for the same attempt are still a contradiction rather than a retry. Each retry runs under its own executor identity (`…_r1`). Found by test: the first implementation re-dispatched correctly and still completed zero trials, because the eval runner treats an existing job_result.json as terminal and replayed the very failure the retry existed to clear. Every executor here resumes its own work from a sealed record, so the attempt number has to reach the adapter and change the run identity. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Serial dispatch made the planned order the executed order, and made a twenty-trial matrix a sixty-five minute wait. `[execution] max_parallel_trials` fans out while still submitting in planned order. For a *paired* design this is arguably better than serial rather than a compromise: both arms of a block now meet the same machine in the same minute, which removes temporal drift instead of adding it. What it does add is contention, which is why the reducer keeps computing fairness from recorded dispatch and start times rather than trusting the setting -- the run this landed for observed a start-order bias of 0.021 against a declared limit of 0.25. The knob is declared in the spec and carried in the plan, so a report can compare intent against what happened. Serial stays the default. A port per trial is the load-bearing half. A base config names one container socket, so concurrent trials would have fought over it -- and a collision would not have looked like an error, it would have looked like one arm scoring the other arm's rollouts. Ports derive from the trial id, are asserted distinct across the matrix, and `container.url` joins the reserved paths a spec cannot make a treatment. The launch command's port literal is rewritten too, since moving only the URL leaves the trial talking to nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat: add experiment layer and GEPA correlation
drop_does_not_delete_heartbeat_owned_by_another_pid overwrote the heartbeat with pid=1 while the guard's heartbeat-writer thread was still live. start_owned_guard writes the heartbeat synchronously and then spawns a thread whose first iteration writes it again (tmp + rename), so that rename could land after the test's overwrite and restore the real PID; Drop then saw its own PID, deleted the file, and the assertion failed (4/30 under the default parallel runner on d3c9edd). Setting the stop flag first was not enough (3/20 still failed): a write already past the flag check still lands. The guard now retains the writer's JoinHandle and the test-only stop_heartbeat_writer joins it, so no write can follow it. Production Drop detaches the handle exactly as before and never joins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…a route that does not exist The 'Make the repository-linked target public' step called PATCH /orgs/synth-laboratories/packages/container/workshop-craftax-eval-target, which GitHub's REST API does not expose (packages support get/list/delete/ restore only; visibility is a package-settings action). It 404ed on runs 32094428767 and 32098386835 after the push itself succeeded, failing the job before the receipt artifact was uploaded. Upload the receipt first, then verify the package is anonymously pullable via the ghcr.io token endpoint and a manifest GET on the published digest. If it is private, fail with the exact one-time settings action an org owner must take. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-visibility-step ci(craftax-target): verify anonymous pullability instead of PATCHing a nonexistent route
…hip-flake test(gepa): stop the heartbeat writer before the foreign-pid drop check
docker/gsm8k-eval-target: an eval.target.v1 container for eval.mlx.local-policy.smoke.v1. stage.py vendors gsm8k_world.py from a pinned containers commit and bakes openai/gsm8k @ 740312add88f781978c0658806c59bc2815b9866 into the context through write_snapshot(), which refuses rows that do not reproduce the recorded split digests. target.py samples the host's synth-mlx-rl over the recipe-owned route pinned to the trial's policy_snapshot_id, verifies the served snapshot, scores exact match with containers' parser, and writes parse_mode into the trace and evidence. .github/workflows/publish-gsm8k-eval-target.yml mirrors the Craftax publish workflow (multi-arch push by digest, receipt artifact); the make-public step is continue-on-error because it is known to 404 on this org. Tests: tests/test_gsm8k_eval_target.py (11) — rig vs policy failure, snapshot pin, route/secret refusal, trace parse mode. Base: origin/v0.7 279eaf5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
workflow_dispatch resolves a workflow by file name on the default branch, so the file has to exist here before `gh workflow run … --ref v07/l2-gsm8k-catalog` can run the lane branch's copy (which carries docker/gsm8k-eval-target/). Same shape as publish-craftax-eval-target.yml; the make-public step is continue-on-error because it is known to 404 on this org. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…orkflow ci: GSM8K eval-target publish workflow on the default branch (dispatch prerequisite)
…t (catalog only) image_digest = sha256:1954fb48382590744643a6716a897234eed8a65899297d72a1313e54c7c7ab5d, recorded by publish-gsm8k-eval-target run 32403202526 (tag v0.7-gsm8k-740312ad-containers-9916cd74, linux/amd64 + linux/arm64; the workflow's receipt artifact is kept under docs/receipts/). No version bump: the pin ships in the next package cut. Until then the operator path is `synth-optimizers eval pin --home … --recipe … --digest …` (home.py:write_pin), documented in docs/local-eval.md with fresh-home `eval doctor` receipts. Also: stage.py registers the vendored module in sys.modules before exec (dataclasses under `from __future__ import annotations`), and a local-trial receipt of the built image against a real synth-mlx-rl. Known: the GHCR package is not publicly pullable yet (make-public step "succeeds" without effect, as for the Craftax target); visibility is an org setting. Base: origin/v0.7 279eaf5 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
eval: GSM8K eval target + publish workflow; pin eval.mlx.local-policy.smoke.v1 by digest (L2, catalog-only)
…-main release(v0.7): reconcile main before promotion
release(v0.7): promote v0.7 to main
Port c187e28 onto the production 0.2.16 baseline; preserve its release history and official Containers dependency. Later v0.10 groups supply final validation and packaging.
Port 9de4740; resolve the obsolete main Containers pin in favor of the source candidate pin. Final packaging will replace development dependencies. Only changelog history differs from the source checkpoint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Supersedes the inherited-stack review in #54. This branch starts from the existing production 0.2.16 baseline (already descended from dev) and ports v0.10 into nine explicit implementation groups, plus CI and whitespace-only follow-ups. It preserves the production changelog rather than overwriting main.
Integration groups
Verification
At 6097962, all files match candidate fe571fe exactly except the restored five-line production changelog. Subsequent changes allow build-only CI on this branch and normalize six inherited documentation files; runtime, tests and dependency bytes remain unchanged. Full branch diff whitespace check passes.
Candidate CI passed, including fresh public PyPI dependency installation on macOS arm64 and Linux x86_64: https://github.com/synth-laboratories/optimizers/actions/runs/34413873850 . Integrated-head validation must finish before merge; the new push runs the same suite and both wheel builds.
Boundaries and next promotion