Skip to content

Bug: flaky test — AgentChatFactoryTests.RestoreSubAgentsAsync_ChildTerminalTaskFaults_ParentInitializeCompletes (SubAgents.Count race) #1459

Description

@JoshuaRowePhantom

Unrelated flaky test — observed during checkin for #1458

Failing test: Phantom.Workspaces.Llm.Tests.AgentChatFactoryTests.RestoreSubAgentsAsync_ChildTerminalTaskFaults_ParentInitializeCompletes

Failure message:

Assert.Equal() Failure: Values differ
Expected: 2
Actual:   0
   at Phantom.Workspaces.Llm.Tests.AgentChatFactoryTests.RestoreSubAgentsAsync_ChildTerminalTaskFaults_ParentInitializeCompletes()
   in Phantom.Workspaces.Llm.Core.Tests/AgentChatFactoryTests.cs:line 726

No dump — this is an assertion failure, not a hang/crash.

Reproduction characterization

  • Failed once during the full fast suite (6290 tests, under heavy parallel scheduler load).
  • Passed 6/6 times in isolation (dotnet test --filter FullyQualifiedName~RestoreSubAgentsAsync_ChildTerminalTaskFaults_ParentInitializeCompletes).
  • Classic load-dependent race.

Root cause hypothesis

In Phantom.Workspaces.Llm.Core/AgentChat.cs RestoreSubAgentsAsync (lines ~1382–1408), each restored child is added to the observable subAgentItems collection via a fire-and-forget task on the foreground scheduler:

_ = Task.Factory.StartNew(
    () => this.subAgentItems.Add(stub),
    CancellationToken.None,
    TaskCreationOptions.DenyChildAttach,
    this.foregroundScheduler);

The test synchronizes on WaitForRestoredSubAgentsMarkedTerminalAsync(), which only awaits restoredSubAgentTerminalTasks (the MarkRestoredSubAgentTerminalAsync tasks). It does not await the foreground-scheduler subAgentItems.Add tasks. Under full-suite scheduler contention those Add tasks can still be pending when the test reads lease.AgentChat.SubAgents.Count, so the assertion observes 0 instead of 2.

Proposed solution

Make the completion of the subAgentItems.Add foreground tasks observable and include them in the wait barrier:

  1. Track the Task.Factory.StartNew(... subAgentItems.Add ...) tasks in a list (e.g. restoredSubAgentAddTasks) under a lock, mirroring restoredSubAgentTerminalTasks.
  2. Have WaitForRestoredSubAgentsMarkedTerminalAsync() (or a new WaitForRestoredSubAgentsAddedAsync()) Task.WhenAll those Add tasks in addition to the terminal tasks.
  3. Update the test to await that combined barrier before asserting SubAgents.Count == 2.

Alternatively, flush the foreground scheduler by scheduling a completion sentinel on foregroundScheduler and awaiting it before the assertion.

This is purely a test/synchronization gap in the sub-agent restore path; the production behavior (eventual population on the foreground scheduler) is correct.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions