Skip to content

feat: bundle, training, task run and agent picks understand always_on and user sims (FD-5371) - #565

Merged
sarah-chitty-1 merged 14 commits into
mainfrom
sarahchitty/fd-5371-always-on-plumbing
Nov 13, 2026
Merged

sarah-chitty-1 merged 14 commits into
mainfrom
sarahchitty/fd-5371-always-on-plumbing

Conversation

@sarah-chitty-1

@sarah-chitty-1 sarah-chitty-1 commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

What

Tools that look for "the agent under test" only knew prompt_agent. This makes them find the agent an always_on step runs. It also stops a run-wide agent pick from swapping the agent that plays the user.

Change

  • bundle run --agent and training find the agent an always_on step runs, as they do for prompt_agent. They never pick the user sim's agent. A training row's first message is the step's prompt, else the opening of the user sim the step names.
  • A step's role comes from its class, not its type name. A step "prompts the agent under test" if it is prompt_agent or always_on. It "plays the user" if its class says so (plays_user, added with the user sim's deploy step in feat(user_sim): the built-in user sim for always-on tasks #567). Both checks live in one small module, task_step/roles.py.
  • task run prints an always_on step's response, as it does for prompt_agent.
  • Episodes: a step that plays the user, or an always_on step, refuses an agent played from outside, for now.
  • Agent picks: deploy_agent and install_agent are unchanged from main. The user sim's deploy step (feat(user_sim): the built-in user sim for always-on tasks #567) keeps its own agent itself.

Behaviour change for existing users

None. prompt_agent tasks work exactly as on main, including a run-wide agent pick, which still swaps every agent they deploy.

Tests

  • run_wide_agent_pick_test.py: a run-wide pick still reaches every deploy_agent and install_agent, as on main;
  • a test step that plays the user (plays_user_step.py), used to check the bundle and training solver lookup, played_deploys and the episode rules never treat it as the agent under test;
  • prompt_agent tasks give the same solver and user agents as on main;
  • the CLI output.

Affected folders pass. The 2 known test_mcp_cli_builder failures are on main too.

Linear: FD-5371.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issue was found.

Summary

The PR adds always_on support to solver lookup and CLI output.

  • Bundle and training tools now find the agent that always_on wakes.
  • task run shows the response from each always_on step.

Reviews (24) · Last reviewed commit: "Drop roles.py and plays_user: the user s..." · Reviewed by Greptile

sarah-chitty-1 and others added 3 commits November 9, 2026 00:10
…t in episodes

The always_on step (FD-5364) runs the agent under test like prompt_agent and names its
user sim through a user_sim step. Code that looks only for prompt_agent found no solver
in such a task. Every change keys on the type name, so it needs none of the new classes.

- bundle run --agent and training: solver_steps counts always_on steps and leaves out
  the agent a user_sim step names (its agent_name, else its name); played_deploys and
  training's first prompt count always_on too.
- task run prints an always_on step's prompt response as it does prompt_agent's.
- Episodes: always_on and user_sim refuse an agent played from outside for now, and the
  default played agent can be an always_on step's, so the refusal names the step. The
  registry test allows the two types before their classes are registered.

Nothing changes for tasks without these step types.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…loys a user sim

The run-wide a2a_agent_id override swapped every deploy_agent and install_agent step,
the user sim's too. When a deploy step deploys a user sim (a prompt_agent's
user_agent_name, or the agent a user_sim step names), task run now moves the pick onto
the tested agent's deploy steps as their step_params, which those steps already prefer.
A step's own step_params pick still wins. Tasks without a user sim keep the run-wide pick.

user_sim_agents, in bundle/plan.py, names a task's user sims for solver_steps and the CLI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… user sim's; one rule for CLI and hub

A run-wide user_overrides.a2a_agent_id (task run --a2a-agent-id, a hub run's agent pick) was applied by every
deploy step, including the one that deploys the user sim's agent, so a hub run with an agent pick played the user
with the tested agent. The CLI narrowing added in 010b041 (_narrow_agent_pick, moving the pick into the tested
agent's step_params) didn't cover the hub, and it also changed the version a picked agent ran at: a pick in
step_params runs the agent's latest version, not the deploy step's stored a2a_agent_version.

The rule now lives in deploy_agent (_resolve_agent_ref), and install_agent follows it: the run-wide pick is not
applied to the agent the run's task names as its user simulator (bundle.plan.user_sim_agents: a user_sim step's
agent_name, else its name; a prompt_agent's user_agent_name), when the task also deploys a tested agent
(solver_steps). That agent keeps its stored a2a_agent_id and version; its own step_params still win. The task is
read from the run's recorded instance (task id and version); a run with no recorded instance, or whose task can't
be read (logged), keeps the pick everywhere, as before. Where the pick applies, today's version rule holds (the
run's a2a_agent_version, else the step's stored version).

The CLI narrowing is removed, so --a2a-agent-id stays run-wide in user_overrides and deploy_agent decides.

Behaviour change for prompt_agent tasks: a CLI or hub run with an agent pick, of a task whose deploy_agent (or
install_agent) step deploys the agent a prompt_agent step names as user_agent_name (default "human_agent"), no
longer swaps that agent for the picked one; it runs the agent the task stores. Tasks that deploy no such agent
are unchanged. On the CLI path the tested agent now runs at its deploy step's stored version again, as before
010b041.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 requested a review from a team as a code owner November 9, 2026 18:11
sarah-chitty-1 and others added 3 commits November 9, 2026 10:16
…user agent name keeps its own agent

Task._run replaces seed placeholders before steps run, so a deploy step named <user> runs as, say, linda. The stored
task was read without the seed, so linda never matched the prompt_agent's <user> and the run-wide pick swapped the
user's agent too. Apply the run's seed the same way before comparing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pick; prompt_agent tasks are unchanged

A prompt_agent task's user_agent_name agent takes a run-wide agent pick exactly as on main, so other teams' runs don't
change. The pick is kept off the user's agent only in tasks with a user_sim step (the always-on step's user).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…by step class, not type strings

Per review: the user sim's deploy becomes its own step (deploy_user_sim_agent, #567), whose class says it plays the
user and ignores a run-wide agent pick itself. So deploy_agent and install_agent no longer re-read the stored task
mid-run to keep the user sim's agent out of a pick; their agent ref is back to main's.

- task_step/roles.py: prompts_agent_under_test (prompt_agent, always_on) and plays_user (the step class's
  plays_user, looked up through the registry for a config; False until #567 adds it).
- solver_steps / user_sim_agents, played_deploys, the episode's trainer agents and refusals, training's prompt pick
  and task run's response printing use them. The user_sim type special cases are gone.
- prompt_agent tasks behave as on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 added a commit that referenced this pull request Nov 11, 2026
deploy_user_sim_agent.py, user_sim_versions.py, user_sim_common.py and user_sim.md move to task_steps/agent/, as Edgar
suggested; imports follow. deploy_agent.py moves in its own PR after #565-#567 merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarah-chitty-1 and others added 2 commits November 11, 2026 21:00
…ning as its first message

An always_on (or prompt_agent) step with no prompt of its own takes the user's opening from the user sim
it names, so the training rows read it from that step's opening_prompt instead of coming out empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rahchitty/fd-5371-always-on-plumbing

# Conflicts:
#	src/agent_env/task/episode/rules.py
Comment thread src/agent_env/cli/task/run.py Resolved
sarah-chitty-1 and others added 2 commits November 12, 2026 10:47
…e from a step running alongside it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rahchitty/fd-5371-always-on-plumbing

# Conflicts:
#	tst/unit/bundle/resolve_test.py
Comment thread src/agent_env/training/harness.py Resolved
sarah-chitty-1 and others added 4 commits November 12, 2026 13:53
…such as a briefing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the plays_user fallback

The user sim's deploy step is already listed in CHECKS (deploys an agent), so the fallback that checked any
plays_user step as the user sim's deploy never ran for it and was a second rule for the same step. Every registered
step type is classified explicitly again, as the module says, and the coverage test no longer exempts plays_user
types or always_on, which is registered now. The two episode tests for the user sim use the real
deploy_user_sim_agent step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a list of type names

roles.prompts_agent_under_test matched the type names prompt_agent and always_on. always_on is a PromptAgentTaskStep
subclass, so is_prompt_step, next to the class it is about, asks whether a step is a prompt_agent or a subclass of
one: a step by its class, a step dict (or anything else with a type) by its type in the step registry, as
is_rubrics_verifier does. A later subclass counts without editing a list. The five callers (bundle plan and resolve,
task run, episodes, training) use it; roles.py keeps plays_user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tep names as its user

Which agent plays the user is already in the task: an always_on step's user_sim and a prompt_agent step's
user_agent_name. user_sim_agents, now next to is_prompt_step in prompt_agent.py, reads those fields from steps or step
dicts, so no class needs a plays_user flag:

- solver_steps and played_deploys subtract user_sim_agents (the deploy-type filter already left out
  deploy_user_sim_agent);
- training's first message finds the user sim as the step with that agent_name and an opening_prompt;
- roles.py, DeployUserSimAgentTaskStep.plays_user and the PlaysUser test step go.

Tests that used the test step use the real deploy_user_sim_agent, named by an always_on step's user_sim (or, in the
training test, by the asking prompt_agent's user_agent_name).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sarah-chitty-1
sarah-chitty-1 merged commit 204e9db into main Nov 13, 2026
76 of 78 checks passed
@sarah-chitty-1
sarah-chitty-1 deleted the sarahchitty/fd-5371-always-on-plumbing branch November 13, 2026 00:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants