Unrelated flaky test — observed during checkin for #1458
Failing test: Phantom.Workspaces.Llm.Tests.AgentChatFactoryTests.RestoreSubAgentsAsync_ChildTerminalTaskFaults_ParentInitializeCompletes
Failure message:
Assert.Equal() Failure: Values differ
Expected: 2
Actual: 0
at Phantom.Workspaces.Llm.Tests.AgentChatFactoryTests.RestoreSubAgentsAsync_ChildTerminalTaskFaults_ParentInitializeCompletes()
in Phantom.Workspaces.Llm.Core.Tests/AgentChatFactoryTests.cs:line 726
No dump — this is an assertion failure, not a hang/crash.
Reproduction characterization
- Failed once during the full fast suite (6290 tests, under heavy parallel scheduler load).
- Passed 6/6 times in isolation (
dotnet test --filter FullyQualifiedName~RestoreSubAgentsAsync_ChildTerminalTaskFaults_ParentInitializeCompletes).
- Classic load-dependent race.
Root cause hypothesis
In Phantom.Workspaces.Llm.Core/AgentChat.cs RestoreSubAgentsAsync (lines ~1382–1408), each restored child is added to the observable subAgentItems collection via a fire-and-forget task on the foreground scheduler:
_ = Task.Factory.StartNew(
() => this.subAgentItems.Add(stub),
CancellationToken.None,
TaskCreationOptions.DenyChildAttach,
this.foregroundScheduler);
The test synchronizes on WaitForRestoredSubAgentsMarkedTerminalAsync(), which only awaits restoredSubAgentTerminalTasks (the MarkRestoredSubAgentTerminalAsync tasks). It does not await the foreground-scheduler subAgentItems.Add tasks. Under full-suite scheduler contention those Add tasks can still be pending when the test reads lease.AgentChat.SubAgents.Count, so the assertion observes 0 instead of 2.
Proposed solution
Make the completion of the subAgentItems.Add foreground tasks observable and include them in the wait barrier:
- Track the
Task.Factory.StartNew(... subAgentItems.Add ...) tasks in a list (e.g. restoredSubAgentAddTasks) under a lock, mirroring restoredSubAgentTerminalTasks.
- Have
WaitForRestoredSubAgentsMarkedTerminalAsync() (or a new WaitForRestoredSubAgentsAddedAsync()) Task.WhenAll those Add tasks in addition to the terminal tasks.
- Update the test to await that combined barrier before asserting
SubAgents.Count == 2.
Alternatively, flush the foreground scheduler by scheduling a completion sentinel on foregroundScheduler and awaiting it before the assertion.
This is purely a test/synchronization gap in the sub-agent restore path; the production behavior (eventual population on the foreground scheduler) is correct.
Unrelated flaky test — observed during checkin for #1458
Failing test:
Phantom.Workspaces.Llm.Tests.AgentChatFactoryTests.RestoreSubAgentsAsync_ChildTerminalTaskFaults_ParentInitializeCompletesFailure message:
No dump — this is an assertion failure, not a hang/crash.
Reproduction characterization
dotnet test --filter FullyQualifiedName~RestoreSubAgentsAsync_ChildTerminalTaskFaults_ParentInitializeCompletes).Root cause hypothesis
In
Phantom.Workspaces.Llm.Core/AgentChat.csRestoreSubAgentsAsync(lines ~1382–1408), each restored child is added to the observablesubAgentItemscollection via a fire-and-forget task on the foreground scheduler:The test synchronizes on
WaitForRestoredSubAgentsMarkedTerminalAsync(), which only awaitsrestoredSubAgentTerminalTasks(theMarkRestoredSubAgentTerminalAsynctasks). It does not await the foreground-schedulersubAgentItems.Addtasks. Under full-suite scheduler contention those Add tasks can still be pending when the test readslease.AgentChat.SubAgents.Count, so the assertion observes0instead of2.Proposed solution
Make the completion of the
subAgentItems.Addforeground tasks observable and include them in the wait barrier:Task.Factory.StartNew(... subAgentItems.Add ...)tasks in a list (e.g.restoredSubAgentAddTasks) under a lock, mirroringrestoredSubAgentTerminalTasks.WaitForRestoredSubAgentsMarkedTerminalAsync()(or a newWaitForRestoredSubAgentsAddedAsync())Task.WhenAllthose Add tasks in addition to the terminal tasks.SubAgents.Count == 2.Alternatively, flush the foreground scheduler by scheduling a completion sentinel on
foregroundSchedulerand awaiting it before the assertion.This is purely a test/synchronization gap in the sub-agent restore path; the production behavior (eventual population on the foreground scheduler) is correct.