Repository navigation
hill-climb: multi-objective (Pareto/epsilon-constraint) support + intake objectives config - #686
Merged
Merged
Conversation
skillberry-bot
added
algorithm
Contributor
|
🏷️ Automatic Labeling I've analyzed this pull request and added the following labels:
These labels were selected based on the PR title, description, and changed files. If you believe any labels are incorrect or missing, feel free to adjust them manually. |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Comment on lines
+71
to
+74
| "import sys, pathlib\n" | ||
| "d = pathlib.Path(sys.argv[1])\n" | ||
| "(d / 'cfg.txt').write_text('A B')\n" | ||
| "(d / 'cost.txt').write_text('1.0')\n", |
Comment on lines
+93
to
+95
| "import sys, pathlib\n" | ||
| "d = pathlib.Path(sys.argv[1])\n" | ||
| "(d / 'cost.txt').write_text('4.0')\n", |
Comment on lines
+114
to
+117
| "import sys, pathlib\n" | ||
| "d = pathlib.Path(sys.argv[1])\n" | ||
| "(d / 'cfg.txt').write_text('A B')\n" | ||
| "(d / 'cost.txt').write_text('999.0')\n", # cost would dominate-reject in pareto mode |
… config Extends hill-climb's per-iteration gate call (harness.run_step) to dispatch to cap_evolve.gate's existing mode="pareto" when the project declares objectives/ gate_mode: pareto, mirroring agent-optimize's gate_check.py --mode pareto semantics. hill-climb's own per-iteration data can resolve one secondary objective today -- per-task cost_usd -- via a new _pareto_metrics_kwargs() helper; any other declared objective name is left for gate.py's own ParetoObjectiveError to refuse rather than fabricate a value. Wiring: hill-climb/scripts/run.py gets --objectives (JSON, mirrors gate_check.py's flag); cli.py forwards capevolve.yaml's objectives: block to it only when algorithm_skill == hill-climb and objectives is set, so an existing spec with no objectives key builds a byte-identical argv and gate call. Also adds objectives as a RECOMMENDED intake-interview input (INPUTS.md), defaulting to the standard reward-maximize + cost-minimize pair (gate._DEFAULT_PARETO_OBJECTIVES) when the user wants multi-objective but doesn't name objectives, and updates templates/project/capevolve.yaml's comments to say hill-climb supports gate_mode: pareto natively now. Known integration point: issue #684's sibling item (1/2, a persistent ParetoArchive module at core/cap_evolve/pareto_archive.py for round.py/ agent-optimize) has not landed on main and has no open PR as of this writing -- checked `gh pr list` and the merged #667 history. This wires hill-climb against gate.py's existing single-pairwise mode="pareto" (from follow-up rebase of _pareto_metrics_kwargs/gate_kwargs plumbing, not a redesign. round.py's own agent-optimize gate cascade, screening, merge_search.py, plan_round.py ownership, the dashboard and cost-reduction guidance are out of scope for this PR (owned by sibling #684 workstreams). Tests: core/tests/test_hill_climb_pareto_gate.py (gate dispatch + no-objectives regression), core/tests/test_hill_climb_objectives_cli.py (end-to-end capevolve.yaml -> cli.py -> hill-climb run.py -> gate, via a real toy_calc run), core/tests/test_intake_objectives_default.py (default-objectives parity between gate.py, INPUTS.md and the template). python -m pytest core/tests -q: 1921 passed, 1 skipped, 2 failed (the two known pre-existing test_tau2_airline_eval_cost.py failures, unrelated to this change). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Osher Elhadad <Osher.Elhadad@ibm.com>
OsherElhadad
force-pushed
the
hill-climb-moo-684
branch
from
October 8, 2026 11:44
a837102 to
158f3e7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #684 (item 8 only). Extends hill-climb's per-iteration gate call to support
multi-objective acceptance, and adds
objectivesas anintake-interview configparameter — fully additive, zero behavior change for existing single-objective
configs.
core/cap_evolve/harness.py:run_step's gate call now buildsmetrics_candidate/metrics_current/their stderrs (new_pareto_metrics_kwargs()helper, per-taskcost_usd) and forwards them togate.decide()only whengate_kwargs["mode"] == "pareto". Every other mode's call is byte-identical to before.skills/algorithms/hill-climb/scripts/run.py: new--objectivesflag (JSON, mirrorsgate_check.py's own flag) forwarded intogate_kwargs["objectives"].core/cap_evolve/cli.py: forwardscapevolve.yaml'sobjectives:block tohill-climb's
--objectivesonly whenalgorithm_skill == hill-climbandobjectivesis set — absent key ⇒ no flag added, no behavior change.skills/phases/intake/inputs/INPUTS.md: new RECOMMENDEDobjectivesinput —default is unset (single-objective, today's behavior); if the user wants
multi-objective and doesn't name objectives, default to the standard
reward-maximize + cost-minimize pair (
gate._DEFAULT_PARETO_OBJECTIVES), with theper-objective significance/honesty-cost note (
ParetoObjectiveError).templates/project/capevolve.yaml: comment update —gate_mode: paretois nolonger agent-optimize-only.
Known integration point (read before merging)
Issue #684's item 1/2 (a persistent
ParetoArchivemodule atcore/cap_evolve/pareto_archive.py, plusepsilon_constraintas a gate mode) hasnot landed on
mainand has no open PR as of this writing — checkedgh pr listand the merged-PR history (#667: "gate: multi-objective Paretoacceptance mode", on
main). This PR wires hill-climb againstgate.py'sexisting single-pairwise
mode="pareto"(from #667) instead of duplicating thatwork.
_pareto_metrics_kwargs()+ thegate_kwargsplumbing are structured so thatswapping in the persistent archive later is a follow-up rebase (reuse
ParetoArchive/the realepsilon_constraintmode once it lands), not a redesign.epsilon_constraintitself does not exist as a gate mode anywhere yet, so it is notwired here — a config declaring
gate_mode: epsilon_constrainttoday getsgate.py'sown
unknown gate modeValueError, same as before this PR.Out of scope (owned by sibling #684 workstreams, untouched here):
round.py's ownagent-optimize gate cascade/screening,
merge_search.py,plan_round.py's ownershiplogic, the dashboard, cost-reduction guidance.
Test plan
core/tests/test_hill_climb_pareto_gate.py— gate dispatches to pareto mode(accept on dominance, reject when dominated) when declared; regression test
confirms the pre-existing single-metric call path is unchanged when no
objectives/pareto mode is set.
core/tests/test_hill_climb_objectives_cli.py— end-to-endcapevolve.yaml→cli.py→hill-climb/run.py→ the gate, driven through areal toy_calc run (
cli.main(["run", ...])), confirming the whole chainactually reaches pareto mode (and that an absent
objectives:key never addsthe flag).
core/tests/test_intake_objectives_default.py— pins the documented defaultobjectives (INPUTS.md + the template) against
gate._DEFAULT_PARETO_OBJECTIVES.python -m pytest core/tests -q: 1921 passed, 1 skipped, 2 failed — the twoknown pre-existing
test_tau2_airline_eval_cost.pyfailures (unrelated to thischange: a
cost_sourcelabel mismatch, nothing to do with the gate).Closes partially #684 (item 8 only).
🤖 Generated with Claude Code