Repository navigation
feat(custom-specialist): make the Coordinator own the custom specialist; add HYPERLOOM_CUSTOM_SPECIALIST_ONLY - #1876
Conversation
Code metrics gate: PASSEDJudged: the 24 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
A9 Tuple return (advisory, never fails): 1
Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 3
Already over at the base (backlog, not this change's): 1676 findings in files this change does not touchOutside this change (informational): 76
...and 51 more; the job log lists every row.
Thresholds and sources: |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
39999fc to
6c97d6f
Compare
| patch_owner_phase, | ||
| ) | ||
| from ..collaborator import CoordinatorCollaborator | ||
| from .proposal_bench import CUSTOM_PROPOSAL_DOMAIN, ProposalBench, kb_exposure_of |
| from hashlib import sha1 | ||
| from typing import Any | ||
|
|
||
| from ..collaborator import CoordinatorCollaborator |
|
|
||
| from ..collaborator import CoordinatorCollaborator | ||
| from ..specialists.domains import CUSTOM_SPECIALIST_KEY | ||
| from .dispatch_order import is_operator_work |
| async def _maybe_finish_custom_only_cycle(self) -> None: | ||
| """Under ``--custom-specialist-only``, hint ``skip_to_sweep`` once this cycle's custom work has all run.""" | ||
| from ..specialists.domains import custom_specialist_key | ||
| from ..state.shared_state import ESCALATE_HINT_SKIP_TO_SWEEP |
…st; add --custom-specialist-only
The operator-defined specialist could be starved for a whole session: Orchestration
chose whether and when to dispatch it, any task tagged custom_specialist satisfied
the at-least-once guarantee without running the operator prompt, and its proposals
waited behind other explore grids, integrate_patch tasks and the framework
candidate track before anyone benched them.
- The Coordinator dispatches it once per macro cycle on FRAMEWORK_AGENT entry,
keyed custom-c{cycle}-{prompt hash}; Orchestration is no longer offered the
domain and the gate denies any delegate naming it as domain or tag at ingress.
- Removed the guarantee latches, custom_specialist_dispatched, the tag-based
dispatch marking, the description in the Orchestration prompt and the
--orch-prompt conflict.
- Operator work (the custom specialist and the grid benching its proposals)
dispatches ahead of every other queued task; its grid no longer waits for
other explores or framework_agent_phase_done, and the general bench leaves
custom proposals out.
- A plateau exit from FRAMEWORK_AGENT waits for pending operator work; budget
and terminal exits do not.
- --custom-specialist-only runs FRAMEWORK_AGENT on the custom specialist and the
bench of its proposals and patches alone, then hints skip_to_sweep.
Co-authored-by: Cursor <cursoragent@cursor.com>
…s gates - PolicyDenied moves to policy/denial.py (gate re-exports it) so the new policy/custom_specialist.py, which holds both custom-specialist ingress rules, imports it without a cycle; the policy package no longer imports the phase machine. - The custom/operator bookkeeping moves into phases/proposal_bench.py (ProposalBench, the FrameworkPhase base), with kb_exposure_of. - MachinePhase._next_phase computes the transition and holds a plateau exit for operator work; is_leverage_exit replaces the flag argument. - The --custom-specialist-* flags and SharedState fields get their own helpers in parser.py and bootstrap.py. - Baseline: three entries lowered by --update-baseline. Co-authored-by: Cursor <cursoragent@cursor.com>
…y in both modes Default mode queued it from the phase pump, which runs after orchestration's first turn and after that tick's dispatch, so the specialists orchestration delegated took the research lane first. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
811b726 to
ae6e75c
Compare
…estration turns Every Orchestration intent there is denied, yet its turn held the tick's dispatch back for 10-20 minutes on each entry and after the custom task finished. The custom task it used to extend now runs to the wall-budget cap, still bounded by the phase and session ends; admission keeps the mode base. Co-authored-by: Cursor <cursoragent@cursor.com>
PR #1876 -- feat(custom-specialist): make the Coordinator own the custom specialist; add --custom-specialist-onlyWhat it does: The operator-defined Blocking issues: 4
SKIPPED: V4 -- openprs.txt compared #1032, #1154, #1276, #1675 and #1869 on their first 100 changed files only, so overlap beyond those files was not checked. Checked: specialists/dispatch.py (ensure_custom_specialist, maybe_force_stalled_domain_specialist, record_specialist_result), loop/intent_router.py (handle_intent, handle_delegate, record_policy_denied), loop/dispatcher.py (admission_denial_for_action, _specialist_wall_budget_sec, _drain_completions), loop/time_budget.py, loop/coordinator.py (_tick_once, _reactor_turn), phases/machine.py, phases/machine_state.py (_framework_exit, compute_next_phase, phase budgets), phases/proposal_bench.py, phases/framework.py (on_enter_framework, pump, maybe_autosubmit_specialist_patches), policy/gate.py (validate_dispatched_task, _validate_delegate_body, _validate_propose_action), policy/custom_specialist.py, AGENTS.md. I also ran a tree-wide git grep over src/, docs/ and assets/ for every removed or renamed name. | Ran: repro scripts at head and base (repeated refused dispatch on a 2 h session, custom-only idle, plateau hold with and without |
…cle; pin the FRAMEWORK exit hold A refused admission left no task row, so ensure_custom_specialist re-requested it every tick, and under --custom-specialist-only the phase idled to its budget. The refused key is kept and counts as a finished custom cycle. The plateau hold is now tested through advance_phase_if_needed. Co-authored-by: Cursor <cursoragent@cursor.com>
…NT over to the next phase, KERNEL_AGENT included Co-authored-by: Cursor <cursoragent@cursor.com>
PR #1876 -- feat(custom-specialist): make the Coordinator own the custom specialist; add --custom-specialist-onlyWhat it does: The operator-defined Blocking issues: 1
SKIPPED: V4 -- openprs.txt compared #1032, #1154, #1276, #1675 and #1869 on their first 100 changed files only, so overlap beyond those files was not checked. Checked: the delta b4b8b6c..54518f9 (specialists/dispatch.py ensure_custom_specialist and refused_custom_key, phases/proposal_bench.py _maybe_finish_custom_only_cycle, cli/parser.py help, docs/reference/environment-variables.md, inference_optimizer/SKILL.md, the new tests), every refusal path of IntentRouter.handle_delegate for the custom params (dispatcher.py admission gates, _admit_specialist, the patch preflight), and the description's numbers against the merge base and the head. Tree-wide git grep for refused_custom_key and for stale "SWEEP" wording. | Ran: the previous review's four findings, re-verified at this head, all resolved. On a 2 h session, a refused dispatch now gives one |
PR #1876 -- feat(custom-specialist): make the Coordinator own the custom specialist; add --custom-specialist-onlyWhat it does: The operator-defined Blocking issues: none SKIPPED: V4 -- openprs.txt compared #1032, #1154, #1276, #1675 and #1869 on their first 100 changed files only, so overlap beyond those files was not checked. Checked: the head is unchanged since the previous review (54518f9, same diff), and the description's only remaining mismatch is corrected. It now gives LGTM |
…IALIST_ONLY instead of a CLI flag The env is read where the --custom-specialist-* pair is validated, so a fresh launch persists it and a resume can turn it on, as the flag did; an unreadable value and an env without the pair both exit 2 before the session starts. Co-authored-by: Cursor <cursoragent@cursor.com>
…er the only mode was the CLI flag Co-authored-by: Cursor <cursoragent@cursor.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
PR #1876 -- feat(custom-specialist): make the Coordinator own the custom specialist; add --custom-specialist-onlyWhat it does: The operator-defined Blocking issues: 1
SKIPPED: V4 -- openprs.txt compared #1032, #1154, #1276, #1675 and #1869 on their first 100 changed files only, so overlap beyond those files was not checked. Checked: the delta 54518f9..e354d85 (7fb2f70, which switches only mode from the flag to the env, and e354d85, which updates the docs), covering cli/init.py ( |
…he --extra-env pins are exported main() read it before _run_optimize exported the operator's pins, so a pinned value was silently ignored and skipped the exit-2 checks. Only mode and the fresh launch's custom conflicts now settle right after the pins land; the custom-specialist launch helpers move to cli/custom_specialist.py. Co-authored-by: Cursor <cursoragent@cursor.com>
PR #1876 -- feat(custom-specialist): make the Coordinator own the custom specialist; add --custom-specialist-onlyWhat it does: The operator-defined Blocking issues: none SKIPPED: V4 -- openprs.txt compared #1032, #1154, #1276, #1675 and #1869 on their first 100 changed files only, so overlap beyond those files was not checked. Checked: the delta e354d85..a9e1d54, covering LGTM |
xiaofei-zheng
left a comment
xiaofei-zheng
left a comment
There was a problem hiding this comment.
PR #1876 -- feat(custom-specialist): make the Coordinator own the custom specialist; add --custom-specialist-only
What it does: the operator-defined custom_specialist could go unrun for a whole session, because Orchestration decided when to delegate it while the Coordinator's at-least-once guarantee was latched off by any task merely tagged custom_specialist. This PR gives the Coordinator sole ownership: on_enter_framework queues one dispatch per macro cycle under a custom-c{cycle}-{sha1(prompt)[:8]} key straight into router.handle_delegate, the gate denies every agent-emitted delegate naming the domain or tag at ingress only, dispatch_order gains an operator tier ahead of the phase's dispatch-first kinds, the custom proposals get their own operator_priority explore grid, and a FRAMEWORK_AGENT optimize_no_more_leverage exit is held while that work is pending (budget and terminal exits are not). It also adds HYPERLOOM_CUSTOM_SPECIALIST_ONLY=1, which scopes FRAMEWORK_AGENT to the custom specialist and its bench: Orchestration takes no reactor turn there, discovery / scout / stalled-domain force / predictor / general bench are skipped, the custom task runs under no_lease_extension, and the phase hints skip_to_sweep once that cycle's work is done or refused.
Blocking issues: 1
- [X2] The title names a
--custom-specialist-onlyflag the branch deliberately does not add [verified]
Problem: the title still reads "add --custom-specialist-only", but commit 8 ("turn only mode on with HYPERLOOM_CUSTOM_SPECIALIST_ONLY instead of a CLI flag") replaced that flag with an env var. At head,_add_custom_specialist_argsregisters only--custom-specialist-prompt-fileand--custom-specialist-description(src/hyperloom/inference_optimizer/cli/parser.py:152), and the PR's own test asserts argparse rejects the flag with exit 2 (src/hyperloom/inference_optimizer/tests/test_custom_specialist.py:873). The body is correct; only the title is stale.
Impact: the title is what the release cut and the commit on main carry. An operator following it runshyperloom optimize --custom-specialist-only ...and gets "unrecognized arguments" instead of the mode.
Action: retitle to name the env var, e.g.feat(custom-specialist): make the Coordinator own the custom specialist; add HYPERLOOM_CUSTOM_SPECIALIST_ONLY.
SKIPPED: local test run -- the test suite cannot be collected on this host (cli/kb.py imports fcntl); verification relied on reading the code, a direct import-level check of dispatch_order/domains, and the 12 green CI test shards at a9e1d54.
Checked: specialists/dispatch.py::ensure_custom_specialist; loop/intent_router.py::handle_intent vs handle_delegate; policy/gate.py::_validate_delegate_body / validate_dispatched_task (check_source split) and policy/custom_specialist.py; phases/framework.py::on_enter_framework, pump, _maybe_bench_untested_proposals, _enqueue_bench_grid, maybe_autosubmit_specialist_patches, materialize_candidate; phases/proposal_bench.py::operator_work_pending, _maybe_finish_custom_only_cycle; phases/machine.py::_next_phase / advance_phase_if_needed against machine_state.py::_framework_exit; phases/dispatch_order.py::is_operator_work; loop/dispatcher.py::_specialist_wall_budget_sec / _specialist_deadline; loop/coordinator.py::_tick_once; state/_shared_state/render.py::untested_proposal_rows; state/shared_state.py::from_dict; cli/init.py::_settle_custom_specialist_only, _resolve_run_topology, the resume conflict block, cli/custom_specialist.py; docs/reference/environment-variables.md, docs/conceptual/optimization-loop.md, SKILL.md, .env.template | Ran: import-level check of dispatch_order.in_dispatch_order / is_operator_work and domains.custom_specialist_key; CI at head (12/12 test shards, coverage, code-metrics, import-linter green) | Base: a56028d | Head: a9e1d54
lishuoshuo-amd
changed the title
Title updated to name HYPERLOOM_CUSTOM_SPECIALIST_ONLY; the only blocking finding is resolved at the same head.
PR #1876 -- feat(custom-specialist): make the Coordinator own the custom specialist; add HYPERLOOM_CUSTOM_SPECIALIST_ONLYWhat it does: the operator-defined Blocking issues: none. The one finding of the earlier review -- the title naming a SKIPPED: local test run -- the test suite cannot be collected on the review host ( Checked: specialists/dispatch.py::ensure_custom_specialist (the (cycle, prompt) key, Ran: import-level check of LGTM |
xiaofei-zheng
left a comment
xiaofei-zheng
left a comment
There was a problem hiding this comment.
No blocking issues. The earlier finding (the title naming a --custom-specialist-only flag the branch does not add) is resolved by the retitle at the same head a9e1d54; the diff and body are unchanged since that review. Paths checked and commands run are listed in the preceding comment.
3 participants
The operator-defined
custom_specialistcould go unrun for a whole session. Three dispatch paths overlapped: Orchestration chose whether and when to delegate it; the Coordinator's at-least-once guarantee was held back by any queued task merely taggedcustom_specialist(on a MiniMax-M3 AgentX run a serving task tagged that way waited 4.5 h for the GPU and kept the guarantee from ever firing); and a tag-only task flippedcustom_specialist_dispatchedfor good. Once it did run, its proposals waited behind other explore grids, integrate_patch tasks andframework_agent_phase_donebefore anyone benched them, and a FRAMEWORK_AGENT plateau exit could leave before they ran.This PR makes the custom specialist Coordinator-owned:
on_enter_framework, before Orchestration's first turn, in both modes, idempotency keycustom-c{cycle}-{sha1(prompt)[:8]}(a resume with a new definition gets its own run). It entersrouter.handle_delegatedirectly, after the gate. A dispatch refused at admission (for example a patch-mode run that the phase budget cannot fit) is requested once per (cycle, definition), logged, and not retried within the cycle.custom_specialistas domain or tag (specialist_custom_coordinator_owned); dispatch-time row replay does not deny it, so the Coordinator's own task runs.dispatch_ordergains an operator tier: the custom specialist and an explore markedoperator_prioritydispatch before the phase's dispatch-first kinds and the rest.operator_priority), not gated by other explores orframework_agent_phase_done; the general bench leaves them out.machine.pyholds a FRAMEWORK_AGENT exit onoptimize_no_more_leveragewhile operator work is pending; budget and terminal exits are never held (kept out of the frozen predicate inputs, whose v6 contract fixes FRAMEWORKpending_workto null).HYPERLOOM_CUSTOM_SPECIALIST_ONLY=1(no CLI flag). It is read in_run_optimizeright after the operator's--extra-envpins are exported, soexportand a pin both reach it; persisted instate.json, and on a resume it turns only mode on, never off; a non-boolean value or the env without the pair exits 2 before the session starts. Scoped to FRAMEWORK_AGENT:coordinator._tick_once): everything it could emit is denied, and its turn held the tick's dispatch back by 10-20 min on entry and again after the custom task finished. The ingress denial of its specialist/explore/integrate_patch stays as the backstop. The critic still runs and reviews the custom patch.no_lease_extension, so its wall budget is the existing 4 h cap rather than its mode base (no Orchestration turn is left to extend it), still bounded by the phase and session ends. Admission keeps asking for the mode base only.skip_to_sweeponce the custom task (or its refused admission), its grid and its patch reviews are done; the exit goes to KERNEL_AGENT when that phase is enabled, else SWEEP. Later phases run as usual.test_custom_specialist.pyrewritten around the new contract (123 tests). Every new behaviour test was run on the unmodified code first and failed. These include:test_the_coordinator_custom_task_passes_the_dispatch_time_gate_and_runs(both modes), which drives a real Coordinator fromensure_custom_specialistthrough the dispatcher to the executor;test_a_refused_custom_dispatch_is_requested_once_per_cyclethrough the real router (five calls over two cycles give two denials; five before the fix);test_framework_entry_enqueues_the_custom_specialist_before_any_other_work;test_only_mode_reaches_the_launch_by_export_or_by_extra_env_pinandtest_a_bad_custom_specialist_launch_exits_before_preflight, both throughcli.mainup to_preflight(a pinned1was dropped, and a pinnedmaybeor a pin without the pair reached_preflight, before the fix);test_only_mode_framework_ticks_without_an_orchestration_turn;test_only_mode_custom_specialist_runs_to_the_phase_end_without_an_extension, which also shows a 30-min session still admits it;test_framework_holds_a_leverage_exit_for_queued_custom_work_but_not_a_budget_exitthroughadvance_phase_if_needed, which fails when the hold in_next_phaseis removed.Replaced tests: the guarantee-latch tests are replaced by
test_one_dispatch_per_cycle_whatever_became_of_it,test_a_task_that_only_borrows_the_custom_tag_does_not_hold_the_dispatch_back,test_dispatch_enqueues_one_task_per_cycle_through_the_real_routerand the refused-dispatch test above;test_spawning_a_tag_only_custom_task_marks_it_dispatchedis removed with the field; the "offered when configured" prompt and gate tests are replaced bytest_orchestration_prompt_never_offers_custom_specialistandtest_gate_denies_every_custom_specialist_delegate.Ran, one file at a time:
test_custom_specialist.py,test_auto_bench_proposals.py,test_phase_state_machine.py,test_optimize_phase_exit.py,test_predictor_pump.py,test_knowledge_plane.py,test_policy_gate.py,test_dispatcher_lane_priority.py,test_dispatcher_execution_shutdown.py,test_coordinator_sync_helpers_coverage_unit.py,test_specialist_concurrent_dispatch.py,test_gpu_research_lane_dispatch.py,test_session_time_budget.py; all pass.ruff check/ruff format --checkclean;lint-imports7 contracts kept; the code-metrics gate passes against the base.E2E, Qwen3-8B on MI355X (conc 64, ISL/OSL 1024, glm-5-3,
--no-kernel), one session per mode at an earlier head:operator_prioritygrid and its patch measured, then FRAMEWORK exited onskip_to_sweep13 s after the patch finished, and SWEEP → CLOSE followed.extend_leasecalls. The no-turn andno_lease_extensionchanges address both; they are covered by the tests above, not yet by a rerun.cli/custom_specialist.py(the--custom-specialist-*pair's loading, resume and conflict rules, moved out ofcli/__init__.py),policy/custom_specialist.py(the custom ingress rules and the custom-only scope),policy/denial.py(PolicyDenied, re-exported bygate.py, so the new policy module imports it without a cycle) andphases/proposal_bench.py(the operator-work bookkeeping,FrameworkPhase's base).Line counts against main:
framework.py3200 (main 3204),gate.py1608 (1647),coordinator.py1429 (1429),dispatcher.py1550 (1550),cli/__init__.py2850 (2907),parser.py1191 (main 1185; the helper grouping the two custom flags),advance_phase_if_needed149 (152); the--custom-specialist-*flags and SharedState fields moved into helpers. Baseline: four entries lowered by--update-baseline, none raised. The policy package does not import the phase machine (.importlinterunchanged).custom_specialist, the guarantee's three latches (custom_specialist_dispatched, the queued/running scan, the per-process cycle marker), tag-based dispatch marking and the--orch-promptconflict. Preserved: the custom prompt/description rendering, research-mode fallback when patches cannot be authored, the CLI definition rules, persistence and resume replacement, and the general auto-bench for non-custom proposals (test_auto_bench_proposals.pyunchanged and passing).--custom-specialist-*set, the custom specialist is queued on FRAMEWORK_AGENT entry every macro cycle and starts in the first dispatch, and its proposals are the next thing measured on the GPU after whatever is already running. An operator no longer sees sessions where it never ran or its proposals were never benched; when the budget cannot admit it, one refusal per cycle is logged instead of one per tick. WithHYPERLOOM_CUSTOM_SPECIALIST_ONLY=1the phase runs nothing else and Orchestration takes no turn there. The custom task may run to the phase end (4 h cap). The phase hands over to KERNEL_AGENT when enabled, else SWEEP, when that work is done or refused.custom_specialist(domain or tag) and its prompt no longer lists it;--orch-promptno longer conflicts with the custom flags;custom_specialist_dispatchedis gone fromstate.json(old files load, the key is dropped).