slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Repository	Step Name	Console log	Pull Request
1441816	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66916
1441697	dotnet/runtime	osx-x64 Debug PALTests
1441691	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66919
1441618	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66899
1441580	dotnet/runtime	ios-arm64 Release AllSubsets_NativeAOT_Smoke	Log	dotnet/runtime#128787
1441536	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128147
1441528	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128736
1441499	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128615
2987523	dotnet-performance	Performance maui_scenarios_android_innerloop Default Mono_Debug_InnerLoop_Emulator ubuntu 2204 x64 AndroidEmulator main	Log
1441493	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128777
1441485	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128423
1441472	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128626
1441465	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#125889
1438842	dotnet/aspnetcore	Test: macOS	Log	dotnet/aspnetcore#66412
1441419	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128365
1441439	dotnet/runtime	ios-arm64 Release AllSubsets_Mono_Smoke	Log
1441407	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128782
1441382	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128762
1441396	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#128781
1440879	dotnet/maui	Controls CollectionView	Log	dotnet/maui#33850
1440877	dotnet/maui	Controls Layout	Log	dotnet/maui#35505
1441368	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54512
2987459	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone17 net11.0	Log
1441345	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#126705
1441328	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128780
1441359	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54511
1441319	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128281
1441304	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128463
1441354	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54335
1440872	dotnet/maui	Controls CollectionView	Log	dotnet/maui#29905
1440875	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35553
1439816	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128383
1441297	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128731
1441334	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54393
1441331	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#128780
1440374	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128656
1441309	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#128463
1441267	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#128722
1440691	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#128180
1441289	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#127770
1441285	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54391
1441279	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#128616
1441254	dotnet/sdk	TestBuild: macOS (x64)	Log
1441223	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#128734
1441248	dotnet/runtime	osx-x64 Debug PALTests	Log
1441232	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log
1441229	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log
1441228	dotnet/runtime	osx-x64 Debug PALTests	Log
1441217	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53268
1441215	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54507
1440729	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35670
1441068	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#128775
1440683	dotnet/maui	Controls Layout	Log	dotnet/maui#35663
1441029	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log	dotnet/runtime#128409
2987309	dotnet-roslyn	Build_macOS_Debug	Log
1441013	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#128746
1440999	dotnet/runtime	ios-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#126669
1441025	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#125437
1440990	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log	dotnet/runtime#128408
1441005	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#128774
1440994	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66774
1440981	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54390
1440977	dotnet/runtime	osx-x64 Debug PALTests	Log	dotnet/runtime#128702
1427772	dotnet/runtime	Libraries Test Run release coreclr osx x64 Debug	Log	dotnet/runtime#126570
1440936	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#128481
1440942	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#54502
1440922	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#127690
1440906	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128669
1440871	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35629
1440910	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#128669
1440896	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53950
1440860	dotnet/maui	Controls Shell	Log	dotnet/maui#35669
1440861	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54444
2987120	dotnet-maui	RunOniOS_MauiRelease x64	Log
1440811	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#128384
1440792	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66487
1440774	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#128763
1440777	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#128763
1440771	dotnet/aspnetcore	Tests: macOS	Log
1440605	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#128553
1440597	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx x64 checked	Log	dotnet/runtime#127498
1440685	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35661
1440694	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log	dotnet/runtime#127498
1438994	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#125437
1440563	dotnet/runtime	Libraries Test Run checked coreclr osx arm64 Debug	Log	dotnet/runtime#128684
1440550	dotnet/runtime	Libraries Test Run release coreclr osx arm64 Debug	Log	dotnet/runtime#128746
1440663	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#128147
1440660	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#53945
1440608	dotnet/runtime	osx-x64 Release NativeAOT_Libs	Log	dotnet/runtime#128553
1440603	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54498
1440600	dotnet/runtime	osx-x64 release CdacDumpTest	Log	dotnet/runtime#127498
1440583	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66894
1440572	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#128750
1440578	dotnet/sdk	AoT: macOS (x64)	Log
2986983	dotnet-maui	RunOniOS_MauiDebug x64	Log
2987012	dotnet-roslyn	Build_macOS_Debug	Log
1440553	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#54397
1438142	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66747
1440543	dotnet/sdk	AoT: macOS (x64)	Log	dotnet/sdk#54501
1440529	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54500
Displaying 100 of 1692 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
120	474	1692

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.