Skip to content

agentHost: lean tools + CLI-parity prompt for HydraFusion - #339165

Open
Aashna Garg (aashna) wants to merge 2 commits into
mainfrom
aashna-hydrafusion-agent-host-cost-parity
Open

Aashna Garg (aashna) wants to merge 2 commits into
mainfrom
aashna-hydrafusion-agent-host-cost-parity

Conversation

@aashna

@aashna Aashna Garg (aashna) commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Brings HydraFusion cost in the Agent Host in line with Copilot CLI (microsoft/vscode-internalbacklog#9756).

Changes

  • Default-exclude 42 VS Code-only tools for hydrafusion (59 → 17 tools); user modelCapabilityOverrides still win.
  • Add CLI tone / search / tool-efficiency prompt sections for hydrafusion.
  • Artifact system_reminder: the agent host doesn't inject one (artifact guidance lives only in the tool description), so excluding add_artifact_or_reference already removes it. No change needed.
  • rename_chat reminder: new sessions use deferred (host-generated) titles, which never inject it. Only restored legacy activeAgent sessions do, and the title controller can't see the launcher's per-model tool filters, so that path is left unchanged.

Results (TerminalBench 2.1, MSBench, 3 replicas/arm)

Arm Resolve AIC/task Opus input/req
Control (main) 77.8% 74.9 44.0K
Tools excluded 78.7% 60.2 29.0K
Tools excluded + CLI prompt (this PR) 81.6% 56.4 27.6K
  • The benchmark excluded all 44 tools; ask_user/web_fetch are kept here because the interactive CLI ships them.
  • AIC vs control: 0.75x (95% CI 0.67–0.83); vs CLI prod: 1.01x (0.91–1.12). Resolve delta +3.7 pts (n.s.).
  • Run IDs: control 2026092980577334, 2026092980578812, 2026092980580219 · tools-excluded 2026092980553243, 2026092980554682, 2026092980556244 · this PR 2026092983442733, 2026092983444192, 2026092983445765 · CLI prod 2026092280637698, 2026092280650597, 63925722795614, 63925724792440, 63925728098446

Default-exclude VS Code-only tools for hydrafusion unless a tool filter is configured, and add CLI tone/search/tool-efficiency prompt sections for hydrafusion (microsoft/vscode-internalbacklog#9756).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 1, 2026 19:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The default profile removes tools still required by workspace attachment, interactive input, and automatic chat-title flows.

Review effort: Balanced
Findings: 1 High severity

Open (1)
What changed in this PR

Aligns HydraFusion Agent Host behavior with Copilot CLI by reducing tool-token overhead and applying CLI-oriented prompt guidance.

Changes:

  • Adds HydraFusion’s default excluded-tool profile while preserving user overrides.
  • Registers HydraFusion-specific tone and tool-efficiency prompt sections.
  • Adds tests and contributor documentation.
File Description
common/​copilotCliConfig.ts Defines HydraFusion’s excluded tools.
node/​copilot/​copilotSessionLauncher.ts Applies default exclusions during launch/resume.
node/​copilot/​prompts/​hydraFusionPrompt.ts Adds HydraFusion prompt guidance.
node/​copilot/​prompts/​allPrompts.ts Registers the new prompt contributor.
node/​copilot/​prompts/​AGENTS.md Documents HydraFusion behavior.
test/​node/​copilotSessionLauncher.test.ts Tests tool-filter defaults and overrides.
test/​node/​agentHostPromptRegistry.test.ts Tests prompt matching and sections.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/vs/platform/agentHost/common/copilotCliConfig.ts Outdated
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Base: 1f648872 Current: bdafcbae

No screenshot changes.

The interactive Copilot CLI ships both tools, so keeping them is true parity (42 excluded).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@aashna
Aashna Garg (aashna) marked this pull request as ready for review October 1, 2026 20:14
export const HYDRAFUSION_DEFAULT_EXCLUDED_TOOLS: readonly string[] = [
'usages', 'problems',
'createJupyterNotebook', 'editNotebook', 'runNotebookCell', 'getNotebookSummary', 'readNotebookCellOutput',
'runTask', 'getTaskOutput', 'createAndRunTask',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we turning these tools off? Feels like we're overfitting for evals here.

@bhavyaus Bhavya U (bhavyaus) Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also tool search defers all of these tools. this will break scenarios in Agents window as well.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants