Skip to content

[copilot-opt] Eliminate high-frequency late validation failures from slow make build+test runs #46904

Description

@github-actions

Problem

Multiple sessions run full make build && make test-unit as late validation steps, with single executions taking 233 seconds or more. When these commands time out or fail late in a session (after many prior tool calls), all prior work is wasted and the session must be retried. Additionally, parallel_validation tool calls take up to 181 seconds. Slow validation that occurs late in session flow is the costliest single source of session waste.

Evidence

  • Analysis window: 2026-07-06 to 2026-07-20 UTC
  • Sessions analyzed: 10 (conversation logs available)
  • Key metrics and examples:
    • Session 29722950675: make build && make test-unit took 233 seconds (3 separate occurrences in same session)
    • Session 29722950675: parallel_validation tool calls took 181 seconds each (2 occurrences)
    • Session 29722950675: runtime-tools-secret_scanning took 275 seconds (2 occurrences)
    • 19 slow tool calls (>10s) in session 29722950675 alone vs. 3-7 in better-performing sessions
    • Across all 10 sessions, 95 total slow tool calls detected; sessions with late validation failures have 3-4x the slow call rate

Proposed Change

  1. Move make build validation earlier in the agent flow — run a lightweight build check immediately after code edits rather than only at the agent-report-progress stage.
  2. Split validation: run make fmt and make lint (fast, <10s) after each edit, reserving make test-unit for a single final validation pass.
  3. Add an explicit timeout budget for validation commands: if make test-unit is expected to exceed 120s, the agent should split test targets or defer non-critical tests.
  4. Investigate the parallel_validation and runtime-tools-secret_scanning slow paths — if they are external service calls, add explicit timeouts and fallback handling.

Expected Impact

  • Catch build/lint failures within the first 5 turns instead of turn 50+
  • Reduce wasted session work from late validation failures by an estimated 30-50%
  • Reduce average session duration by eliminating redundant full rebuild+test cycles

Notes

  • Distinct root cause category: late validation strategy / slow tool calls
  • Data quality: 10 of 50 sessions have conversation logs; remaining 40 sessions have no logs available for analysis

Generated by ⚡ Copilot Opt · 33.8 AIC · ⌖ 9.79 AIC · ⊞ 6.8K ·

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions