Repository navigation
Expand file tree
/
Copy pathRun-DebugGauntlet.ps1
More file actions
501 lines (459 loc) · 27 KB
/
Copy pathRun-DebugGauntlet.ps1
File metadata and controls
501 lines (459 loc) · 27 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
#!/usr/bin/env pwsh
<#
.SYNOPSIS
Full autonomous build/test/validation gauntlet for AetherCore.
.DESCRIPTION
One entry point that runs the whole debug loop without a human in it:
0. Guard - the GPU abstraction guard (scripts/check-gpu-abstraction.ps1).
1. Build - compile Editor, GameRuntime and EngineTests (MSVC multi-config).
2. Unit - run EngineTests.exe and parse the doctest summary.
3. Editor - launch Editor.exe under whatever Vulkan validation tier the build
was compiled with, wait (by polling the log, not a fixed sleep)
until the startup scene loads, render a few frames, then stop.
Every Validation-category message is classified and any
error / warning / best-practice advisory fails the phase.
4. Runtime - the same smoke for the shipped game runtime (AetherGame.exe).
If a smoke process crashes (e.g. 0xC0000005) or never reaches the ready
marker, the harness automatically re-runs it under cdb with the Vulkan layer
symbols loaded and captures the faulting stack into the report.
Emits a human-readable summary and a machine-readable report.json, and exits
non-zero if any phase fails, so CI can gate on it.
The Vulkan validation TIER is a compile-time choice in src/engine/Defines.hpp
(VULKAN_CPU_DEBUG / VULKAN_GPU_DEBUG / VULKAN_BEST_PRACTICES); this harness
runs whatever the current build compiled and reports which tier it detected.
.PARAMETER BuildDir
CMake build directory (default: build/vs2022-msvc).
.PARAMETER Config
Multi-config build type: Debug or Release (default: Debug).
.PARAMETER Targets
Targets to build (default: Editor GameRuntime EngineTests). In a build tree
configured with AETHERCORE_BUILD_EDITOR=OFF, Editor is dropped automatically
and the editor smoke is skipped (runtime + tests still run).
.PARAMETER RunSeconds
Frames-worth of extra runtime AFTER the startup scene loads, per smoke
(default: 12). GPU-AV instrumentation is slow; bump this for that tier.
.PARAMETER ReadyTimeoutSeconds
Max wait for the "startup scene loaded" marker before declaring a hang
(default: 60; GPU-AV startup can take ~25s).
.PARAMETER Repack
Delete the built engine.pak/project.pak before building so shaders repack
from source (use after editing shaders or the SPIR-V processor).
.PARAMETER ReportPath
Where to write the JSON report (default: <BuildDir>/gauntlet-report.json).
.PARAMETER SkipGuard / -SkipBuild / -SkipUnit / -SkipEditor / -SkipRuntime
Skip individual phases.
.PARAMETER CI
Hosted-runner mode: skips the Editor and Runtime smokes (a hosted runner has no
GPU or display). Explicit only - the manual CI workflow passes it. It is NOT
inferred from $env:CI: agent and tool shells set that variable, which silently
downgraded the local full run to build+unit while still printing GAUNTLET PASSED.
.EXAMPLE
./scripts/Run-DebugGauntlet.ps1
./scripts/Run-DebugGauntlet.ps1 -Config Release -Repack
./scripts/Run-DebugGauntlet.ps1 -CI # build + unit only
./scripts/Run-DebugGauntlet.ps1 -SkipRuntime -RunSeconds 25
#>
param(
# Reuse the dev tree instead of the separate Debug one. Same validation coverage (the
# smokes force --validation, and the layer is compiled into both dev configs), and it
# skips a second full build of the whole engine - which is what makes the default mode
# expensive after any widely-included header changes. The tradeoff is real and worth
# stating: RelWithDebInfo drops AE_ASSERT, so this trades assert coverage for minutes.
# Use it as the routine gate; run the default before merging.
[switch]$Fast,
[string]$BuildDir = "",
[ValidateSet("Debug", "RelWithDebInfo", "Release")]
[string]$Config = "",
[string[]]$Targets = @("Editor", "GameRuntime", "EngineTests", "aether-ctl"),
[int]$RunSeconds = 12,
[int]$ReadyTimeoutSeconds = 60,
[switch]$Repack,
[string]$ReportPath = "",
[switch]$SkipBuild,
[switch]$SkipUnit,
[switch]$SkipEditor,
[switch]$SkipRuntime,
[switch]$SkipGuard,
[switch]$CI
)
$ErrorActionPreference = "Stop"
$RepoRoot = Split-Path -Parent $PSScriptRoot
. "$PSScriptRoot/Qualification.Common.ps1"
# -Fast picks the dev tree; an explicit -BuildDir/-Config still wins over both.
if (-not $BuildDir) { $BuildDir = if ($Fast) { "build/default" } else { "build/vs2022-msvc" } }
if (-not $Config) { $Config = if ($Fast) { "RelWithDebInfo" } else { "Debug" } }
$BuildRoot = if ([System.IO.Path]::IsPathRooted($BuildDir)) { $BuildDir } else { Join-Path $RepoRoot $BuildDir }
# Where CMake actually put the binaries.
#
# Multi-config generators (Visual Studio) nest them under <config>/; single-config ones
# (Ninja) put them straight in the target directory. Hard-coding the config subdirectory
# meant every phase reported "not found" against a Ninja tree in which everything was
# built and passing - the same assumption that once staged the app bundle one level below
# its own executable.
#
# CMAKE_CONFIGURATION_TYPES is the authority: CMake writes it into the cache only for a
# multi-config generator, which is exactly the distinction being made here.
$IsMultiConfig = $false
$CacheFile = Join-Path $BuildRoot "CMakeCache.txt"
if (Test-Path $CacheFile) {
$IsMultiConfig = [bool](Select-String -LiteralPath $CacheFile -Pattern '^CMAKE_CONFIGURATION_TYPES:' -Quiet)
}
# Join a build-tree path whose <config> segment is written as {config}, resolving that
# segment to the configuration directory or to nothing.
function Join-BuildPath([string]$relative) {
$resolved = if ($IsMultiConfig) { $relative -replace '\{config\}', $Config } else { $relative -replace '\{config\}/', '' -replace '/\{config\}', '' }
return Join-Path $BuildRoot $resolved
}
if (-not $ReportPath) { $ReportPath = Join-Path $BuildRoot "gauntlet-report.json" }
# Runtime-only tolerance. A tree configured with AETHERCORE_BUILD_EDITOR=OFF has no
# Editor target, so asking cmake to build it (or smoking its exe) would fail for a
# reason that is not a regression. The build directory's cache is the authority;
# a not-yet-configured tree keeps the default all-targets behaviour.
$EditorDisabled = $false
if (Test-Path $CacheFile) {
$EditorDisabled = [bool](Select-String -LiteralPath $CacheFile -Pattern '^AETHERCORE_BUILD_EDITOR:BOOL=OFF' -Quiet)
}
if ($EditorDisabled -and $Targets -contains "Editor") {
$Targets = @($Targets | Where-Object { $_ -ne "Editor" })
Write-Host " AETHERCORE_BUILD_EDITOR=OFF in this build tree - dropped Editor from the target list." -ForegroundColor DarkYellow
}
# -CI is explicit on purpose; see the .PARAMETER note. Do not re-add $env:CI sniffing.
if ($CI) { $SkipEditor = $true; $SkipRuntime = $true }
# Fresh per-run user state prevents stale log evidence and leaves real preferences alone.
$UserState = Join-Path $RepoRoot "logs/gauntlet-state/$([guid]::NewGuid().ToString('N'))"
New-Item -ItemType Directory -Force $UserState | Out-Null
$LogDir = Join-Path $UserState 'logs'
$ExecutableSuffix = if ($env:OS -eq 'Windows_NT') { '.exe' } else { '' }
$portProbe = [Net.Sockets.UdpClient]::new(0)
$SmokePort = $portProbe.Client.LocalEndPoint.Port
$portProbe.Dispose()
# --- Console helpers -----------------------------------------------------
function Write-Head($text) { Write-Host "`n=== $text ===" -ForegroundColor Cyan }
function Write-Ok($text) { Write-Host " [PASS] $text" -ForegroundColor Green }
function Write-Bad($text) { Write-Host " [FAIL] $text" -ForegroundColor Red }
function Write-Skip($text) { Write-Host " [SKIP] $text" -ForegroundColor DarkGray }
function Write-Note($text) { Write-Host " $text" -ForegroundColor DarkYellow }
$phases = [System.Collections.Generic.List[object]]::new()
function Add-Phase($obj) { $phases.Add([pscustomobject]$obj) }
# Strip ANSI colour codes the engine logger emits, so pattern matching is clean.
function Remove-Ansi([string[]]$lines) {
return $lines | ForEach-Object { $_ -replace "$([char]27)\[[0-9;]*m", "" }
}
# --- cdb (crash triage) --------------------------------------------------
function Get-Cdb {
$candidates = @(
"${env:ProgramFiles(x86)}\Windows Kits\10\Debuggers\x64\cdb.exe"
"${env:ProgramFiles}\Windows Kits\10\Debuggers\x64\cdb.exe"
"${env:ProgramFiles(x86)}\Windows Kits\11\Debuggers\x64\cdb.exe"
)
foreach ($c in $candidates) { if (Test-Path $c) { return $c } }
return $null
}
function Get-SymbolPath {
$sym = "srv*$env:LOCALAPPDATA\dbg\sym*https://msdl.microsoft.com/download/symbols"
$vkBin = if ($env:VULKAN_SDK) { Join-Path $env:VULKAN_SDK "Bin" } else { $null }
if (-not ($vkBin -and (Test-Path $vkBin))) {
$vk = Get-ChildItem "C:\VulkanSDK" -Directory -ErrorAction SilentlyContinue | Sort-Object Name -Descending | Select-Object -First 1
if ($vk) { $vkBin = Join-Path $vk.FullName "Bin" }
}
if ($vkBin -and (Test-Path $vkBin)) { $sym += ";$vkBin" }
return $sym
}
# Attach cdb NON-INVASIVELY to a still-running (hung) process, dump every
# thread's stack, and return the frames in our own code (the interesting ones for
# a deadlock/livelock). Attaching + dumping returns immediately - unlike the
# crash path below, it must NEVER run 'g', which would block forever on a process
# that isn't going to fault.
function Get-HangStacks([int]$ProcId) {
$cdb = Get-Cdb
if (-not $cdb) { return @("<cdb not installed - install the Windows SDK Debuggers for auto hang triage>") }
$out = Join-Path ([System.IO.Path]::GetTempPath()) ("gauntlet_hang_" + [System.IO.Path]::GetRandomFileName() + ".txt")
$sym = Get-SymbolPath
& $cdb -p $ProcId -y $sym -c "~*k 24; qd" > $out 2>&1
$lines = Get-Content $out -ErrorAction SilentlyContinue
Remove-Item $out -Force -ErrorAction SilentlyContinue
if (-not $lines) { return @("<cdb produced no output attaching to pid $ProcId>") }
$frames = @()
foreach ($line in $lines) {
# Keep frames in engine/app code or the thread/sync primitives that reveal
# who is waiting on whom; skip pure CRT/OS noise and module-load lines.
if ($line -match "!aether::|!main\b|ThreadLoop|WaitUntil|RenderGraph|ExecuteRenderFrame|RunFrameLoop|condition_variable::wait|Semaphore::Acquire") {
$site = ($line -split "\s{2,}")[-1].Trim()
if ($site -and $site -match "!" -and $frames -notcontains $site) { $frames += $site }
}
}
if ($frames.Count -eq 0) { return @("<attached but no engine frames - see all-thread dump>") }
return ($frames | Select-Object -First 24)
}
# Re-run a crashed exe under cdb and return the faulting stack (module!symbol
# frames). Returns $null if cdb is unavailable or no access violation reproduced.
function Get-CrashStack([string]$Exe, [string]$WorkDir) {
$cdb = Get-Cdb
if (-not $cdb) { return @("<cdb not installed - install the Windows SDK Debuggers for auto crash triage>") }
$out = Join-Path ([System.IO.Path]::GetTempPath()) ("gauntlet_cdb_" + [System.IO.Path]::GetRandomFileName() + ".txt")
$sym = Get-SymbolPath
$cmds = "sxe av; g; .echo ===FAULT===; kb 25; q"
Push-Location $WorkDir
try {
& $cdb -y $sym -c $cmds $Exe > $out 2>&1
} finally {
Pop-Location
}
$lines = Get-Content $out -ErrorAction SilentlyContinue
Remove-Item $out -Force -ErrorAction SilentlyContinue
if (-not $lines) { return @("<cdb produced no output>") }
# Take the frames after the last ===FAULT=== marker.
$faultIdx = -1
for ($i = 0; $i -lt $lines.Count; $i++) { if ($lines[$i] -match "===FAULT===") { $faultIdx = $i } }
if ($faultIdx -lt 0) { return @("<no access violation reproduced under cdb>") }
$frames = @()
for ($i = $faultIdx + 1; $i -lt $lines.Count -and $frames.Count -lt 20; $i++) {
$line = $lines[$i]
if ($line -match "!" -and $line -notmatch "^ModLoad|^\s*\*\*\*") {
# Keep just the "module!symbol+offset" call site (last column).
$site = ($line -split "\s{2,}")[-1].Trim()
if ($site -and $site -match "!") { $frames += $site }
}
}
if ($frames.Count -eq 0) { return @("<fault captured but no symbolic frames - see cdb>") }
return $frames
}
# --- Validation-finding classification -----------------------------------
# A "finding" that fails a smoke: any ERRO (engine or validation), any
# Validation-category WARN, or any Validation-category INFO advisory (perf /
# best-practice). Benign non-validation warnings (e.g. the scene-format
# migration note) are reported but do not fail.
function Measure-Findings([string[]]$logLines) {
return Measure-ValidationFindings $logLines
}
# --- Phase 0: Static guards ----------------------------------------------
# The architectural check that used to run on a hosted runner. It costs seconds on the
# machine the code was written on, so there is no reason for it to live anywhere else.
function Invoke-Guard {
Write-Head "Phase 0: GPU abstraction guard"
$guardScript = Join-Path $PSScriptRoot "check-gpu-abstraction.ps1"
if (-not (Test-Path $guardScript)) {
Write-Bad "check-gpu-abstraction.ps1 not found at $guardScript"
Add-Phase @{ name = "guard"; status = "fail"; detail = "guard script missing" }
return $false
}
# The guard shells out to ripgrep for every check. Without it each rg call errors and
# the checks quietly pass on nothing, so refuse rather than report a false clean.
if (-not (Get-Command rg -ErrorAction SilentlyContinue)) {
Write-Bad "ripgrep (rg) is not on PATH - the guard cannot run"
Add-Phase @{ name = "guard"; status = "fail"; detail = "ripgrep not installed" }
return $false
}
Push-Location $RepoRoot
try { & $guardScript; $exit = $LASTEXITCODE }
finally { Pop-Location }
$ok = ($exit -eq 0)
if ($ok) { Write-Ok "No new GPU abstraction leaks" }
else { Write-Bad "GPU abstraction guard failed (exit $exit)" }
Add-Phase @{ name = "guard"; status = if ($ok) { "pass" } else { "fail" } }
return $ok
}
# --- Phase 1: Build ------------------------------------------------------
function Invoke-Build {
Write-Head "Phase 1: Build ($Config)"
$sw = [System.Diagnostics.Stopwatch]::StartNew()
if ($Repack) {
Get-ChildItem (Join-Path $BuildRoot "data") -Filter "*.pak*" -ErrorAction SilentlyContinue | Remove-Item -Force -ErrorAction SilentlyContinue
Write-Note "Removed built paks - shaders will repack."
}
$args = @("--build", $BuildRoot, "--config", $Config, "--target") + $Targets
# CMake prints status (e.g. "volk: using Vulkan_INCLUDE_DIRS...") on stderr; under
# $ErrorActionPreference="Stop" the 2>&1 merge turns those into terminating
# NativeCommandErrors. Success is judged by $LASTEXITCODE + error regex below.
$ErrorActionPreference = "Continue"
$output = & cmake @args 2>&1
$ErrorActionPreference = "Stop"
$exit = $LASTEXITCODE
$sw.Stop()
$compileErrors = @($output | Where-Object { $_ -match "error C[0-9]|error LNK|error MSB|: error " } | Select-Object -First 25)
$ok = ($exit -eq 0) -and ($compileErrors.Count -eq 0)
if ($ok) { Write-Ok "Built $($Targets -join ', ') in $($sw.Elapsed.TotalSeconds.ToString('F1'))s" }
else {
Write-Bad "Build failed (exit $exit)"
$compileErrors | ForEach-Object { Write-Note $_ }
}
Add-Phase @{ name = "build"; status = if ($ok) { "pass" } else { "fail" }; durationSec = [math]::Round($sw.Elapsed.TotalSeconds, 1); targets = $Targets; errors = $compileErrors }
return $ok
}
# --- Phase 2: Unit tests -------------------------------------------------
function Invoke-UnitTests {
$suiteReport = Join-Path (Split-Path ([IO.Path]::GetFullPath($ReportPath))) 'ctest-qualification.json'
& pwsh -NoProfile -File "$PSScriptRoot/Invoke-CTestQualification.ps1" -BuildDir $BuildRoot -Config $Config -ReportPath $suiteReport
$ok = $LASTEXITCODE -eq 0
$detail = Get-Content -LiteralPath $suiteReport -Raw | ConvertFrom-Json
Add-Phase @{ name = 'unit'; status = if ($ok) { 'pass' } else { 'fail' }; testCases = $detail.testCases; passed = $detail.passed; failed = $detail.failed; suites = $detail.suites; report = $suiteReport }
return $ok
}
# --- Smoke (shared by editor + runtime) ----------------------------------
function Invoke-Smoke([string]$Name, [string]$ExePath, [string]$LogName, [string]$ExeArgs = "") {
Write-Head "Phase: $Name smoke"
$exe = Join-BuildPath $ExePath
if (-not (Test-Path $exe)) {
Write-Bad "$Name executable not found at $exe"
Add-Phase @{ name = $Name; status = "fail"; detail = "executable missing: $exe" }
return $false
}
$log = Join-Path $LogDir $LogName
if (Test-Path $log) { Remove-Item $log -Force }
# Launch from the build root: shaders:// and engine.pak resolve relative to CWD.
# Ask for validation explicitly rather than inheriting the config's default. The layer
# is compiled into BOTH dev configs (VULKAN_CPU_DEBUG is gated on AE_DEV_TOOLING) but
# only defaults on in Debug - so without this the same harness pointed at a
# RelWithDebInfo tree runs with no validation at all and still reports a clean pass.
$ExeArgs = ("$ExeArgs --validation").Trim()
$start = @{ FilePath = $exe; ArgumentList = $ExeArgs; WorkingDirectory = $BuildRoot; PassThru = $true }
$start.Environment = @{ AETHER_USER_DATA_DIR = $UserState; AETHER_CONTROL_PORT = 'off' }
if ($env:OS -eq 'Windows_NT') { $start.WindowStyle = 'Hidden' }
$proc = Start-Process @start
# Condition-based readiness: poll the log for the startup-scene marker rather
# than sleeping a fixed amount. Fail fast if the process dies first.
$ready = $false; $crashedEarly = $false
$deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds)
while ((Get-Date) -lt $deadline) {
if ($proc.HasExited) { $crashedEarly = $true; break }
if (Test-Path $log) {
$content = Remove-Ansi (Get-Content $log -ErrorAction SilentlyContinue)
if ($content | Where-Object { $_ -match "Startup scene .* loaded" }) { $ready = $true; break }
}
Start-Sleep -Milliseconds 500
}
$crash = $null
if ($crashedEarly -or -not $ready) {
if ($proc.HasExited) {
# Crashed before reaching the ready marker: re-run under cdb to catch
# the access violation deterministically.
$exitCode = "0x{0:X8}" -f ($proc.ExitCode -band 0xFFFFFFFF)
Write-Bad "$Name crashed before ready (exit $exitCode) - triaging under cdb..."
$stack = Get-CrashStack -Exe $exe -WorkDir $BuildRoot
}
else {
# Still running but never signalled ready: a hang/deadlock. Attach to
# the LIVE process and dump thread stacks (do not relaunch-and-'g').
$exitCode = "hang (no ready marker in ${ReadyTimeoutSeconds}s)"
Write-Bad "$Name hung - attaching to pid $($proc.Id) to dump live thread stacks..."
$stack = Get-HangStacks -ProcId $proc.Id
Stop-Process -Id $proc.Id -Force -ErrorAction SilentlyContinue
}
$stack | Select-Object -First 10 | ForEach-Object { Write-Note $_ }
$crash = [pscustomobject]@{ exitCode = $exitCode; stack = $stack }
$logLines = if (Test-Path $log) { Remove-Ansi (Get-Content $log -ErrorAction SilentlyContinue) } else { @() }
$f = Measure-Findings $logLines
Add-Phase @{ name = $Name; status = "fail"; reachedReady = $false; crash = $crash; validationTier = $f.Tier; findings = @{ errors = $f.Errors.Count; warnings = $f.ValWarn.Count; advisories = $f.ValInfo.Count } }
return $false
}
# Rendered to the startup scene; let it run a few more frames to exercise the
# render graph and surface per-frame validation, then stop cleanly.
Start-Sleep -Seconds $RunSeconds
$liveCrash = $proc.HasExited
$liveExit = if ($liveCrash) { "0x{0:X8}" -f ($proc.ExitCode -band 0xFFFFFFFF) } else { $null }
$client = Join-BuildPath "tools/control-client/{config}/aether-ctl$ExecutableSuffix"
$shutdown = if ($Name -eq 'editor' -and (Test-Path -LiteralPath $client)) { Stop-QualifiedProcess $proc $client $SmokePort 30 } else { Stop-QualifiedProcess -Process $proc -TimeoutSeconds 30 }
Start-Sleep -Milliseconds 500
$logLines = Remove-Ansi (Get-Content $log -ErrorAction SilentlyContinue)
$f = Measure-Findings $logLines
if ($liveCrash) {
Write-Bad "$Name crashed mid-run (exit $liveExit) - triaging..."
$stack = Get-CrashStack -Exe $exe -WorkDir $BuildRoot
$stack | Select-Object -First 8 | ForEach-Object { Write-Note $_ }
$crash = [pscustomobject]@{ exitCode = $liveExit; stack = $stack }
}
# A smoke that cannot confirm the layer was loaded proves nothing: zero findings and
# zero coverage look identical from here. Treat an undetected tier as a failure rather
# than reporting the most reassuring possible result for the least informative run.
$tierUnknown = $f.Tier -eq "unknown"
if ($tierUnknown) {
Write-Bad "$Name ran without the validation layer (tier: unknown) - findings from this run mean nothing."
}
$shutdownLogOk = $true
try { Assert-ShutdownLog $logLines } catch { $shutdownLogOk = $false; Write-Bad $_.Exception.Message }
$ok = (-not $liveCrash) -and ($f.Failing -eq 0) -and (-not $tierUnknown) -and ($shutdown.status -eq 'pass') -and $shutdownLogOk
if ($ok) {
Write-Ok "$Name clean - tier: $($f.Tier); 0 validation findings"
if ($f.OtherWarn.Count -gt 0) { Write-Note "($($f.OtherWarn.Count) benign non-validation warning(s), e.g. scene-format migration)" }
}
else {
if ($f.Errors.Count) { Write-Bad "$($f.Errors.Count) error line(s):"; $f.Errors | Select-Object -First 6 | ForEach-Object { Write-Note $_ } }
if ($f.ValWarn.Count) { Write-Bad "$($f.ValWarn.Count) validation warning(s):"; $f.ValWarn | Select-Object -First 6 | ForEach-Object { Write-Note $_ } }
if ($f.ValInfo.Count) { Write-Bad "$($f.ValInfo.Count) best-practice/perf advisory(ies):"; $f.ValInfo | Select-Object -First 6 | ForEach-Object { Write-Note $_ } }
}
Add-Phase @{
name = $Name; status = if ($ok) { "pass" } else { "fail" }; reachedReady = $true
shutdown = $shutdown; shutdownLogVerified = $shutdownLogOk
validationTier = $f.Tier
findings = @{ errors = $f.Errors.Count; warnings = $f.ValWarn.Count; advisories = $f.ValInfo.Count; benignWarnings = $f.OtherWarn.Count }
messages = @($f.Errors + $f.ValWarn + $f.ValInfo | Select-Object -First 30)
crash = $crash
}
return $ok
}
# --- Run -----------------------------------------------------------------
Write-Host "AetherCore Debug Gauntlet" -ForegroundColor White
Write-Host " build: $BuildRoot ($Config) | mode: $(if ($CI) { 'CI (build+unit)' } elseif ($Fast) { 'fast (dev tree, no asserts)' } else { 'full' })" -ForegroundColor DarkGray
$results = @{}
$overall = $true
# Independent of the build, so it runs first and never short-circuits it.
if ($SkipGuard) { Write-Head "Phase 0: GPU abstraction guard"; Write-Skip "skipped"; Add-Phase @{ name = "guard"; status = "skip" } }
else { $results.guard = Invoke-Guard; if (-not $results.guard) { $overall = $false } }
if ($SkipBuild) { Write-Head "Phase 1: Build"; Write-Skip "skipped"; Add-Phase @{ name = "build"; status = "skip" } }
else { $results.build = Invoke-Build; if (-not $results.build) { $overall = $false } }
# A broken build makes every later phase meaningless - short-circuit.
$buildOk = $SkipBuild -or $results.build
if (-not $buildOk) {
Write-Head "Aborting: build failed"
}
else {
if ($SkipUnit) { Write-Head "Phase 2: Unit tests"; Write-Skip "skipped"; Add-Phase @{ name = "unit"; status = "skip" } }
else { $results.unit = Invoke-UnitTests; if (-not $results.unit) { $overall = $false } }
if ($SkipEditor) { Write-Head "Phase: editor smoke"; Write-Skip $(if ($CI) { "skipped (CI: no GPU/display)" } else { "skipped" }); Add-Phase @{ name = "editor"; status = "skip" } }
elseif ($EditorDisabled) {
Write-Head "Phase: editor smoke"
Write-Skip "skipped (AETHERCORE_BUILD_EDITOR=OFF - runtime-only tree, no Editor to smoke)"
Add-Phase @{ name = "editor"; status = "skip"; detail = "editor not built (AETHERCORE_BUILD_EDITOR=OFF)" }
}
# The editor is always project-scoped now: pass --project (as F5 does), else it
# exits requesting one. (Launching the Launcher would spawn a separate process the
# smoke can't track, so drive the Editor directly.)
# Editor.exe's OUTPUT dir follows the CMakeLists location (src/app), not the
# source tree move - only the sources live in src/editor.
else { $results.editor = Invoke-Smoke -Name "editor" -ExePath "src/app/{config}/Editor$ExecutableSuffix" -LogName "Editor.log" -ExeArgs "--project `"$(Join-Path $RepoRoot 'projects/TestingProject')`" --agent --background --agent-name gauntlet --control-port $SmokePort"; if (-not $results.editor) { $overall = $false } }
if ($SkipRuntime) { Write-Head "Phase: runtime smoke"; Write-Skip $(if ($CI) { "skipped (CI: no GPU/display)" } else { "skipped" }); Add-Phase @{ name = "runtime"; status = "skip" } }
# Point the runtime at the same project the editor smoke uses. Without this it resolves
# its project from the build tree, and a build tree that has ever been packed into has a
# data/project.pak - which the runtime correctly reads as "published package, use the
# baked settings" - so it boots an EMPTY world, never logs the ready marker, and the
# smoke reports a hang for a process that is perfectly healthy. AETHER_PROJECT_DIR is the
# documented override and beats that check.
else {
$env:AETHER_PROJECT_DIR = Join-Path $RepoRoot 'projects/TestingProject'
try { $results.runtime = Invoke-Smoke -Name "runtime" -ExePath "src/app/{config}/AetherGame$ExecutableSuffix" -LogName "AetherGame.log" }
finally { Remove-Item Env:\AETHER_PROJECT_DIR -ErrorAction SilentlyContinue }
if (-not $results.runtime) { $overall = $false }
}
}
# --- Report --------------------------------------------------------------
$report = [pscustomobject]@{
timestamp = (Get-Date).ToString("o")
repoRoot = $RepoRoot
buildDir = $BuildRoot
userState = $UserState
config = $Config
mode = if ($CI) { "ci" } else { "full" }
overall = if (-not $overall) { 'FAIL' } elseif (@($phases | Where-Object { $_.status -ne 'pass' }).Count) { 'PARTIAL' } else { 'PASS' }
qualification = Get-QualificationStatus $phases @('guard','build','unit','editor','runtime')
phases = $phases
}
New-Item -ItemType Directory -Force -Path (Split-Path -Parent $ReportPath) | Out-Null
$report | ConvertTo-Json -Depth 8 | Set-Content -Path $ReportPath -Encoding UTF8
Write-Head "Summary"
foreach ($p in $phases) {
$colour = switch ($p.status) { "pass" { "Green" } "fail" { "Red" } default { "DarkGray" } }
Write-Host (" {0,-9} {1}" -f $p.name, $p.status.ToUpper()) -ForegroundColor $colour
}
Write-Host "`n report: $ReportPath" -ForegroundColor DarkGray
if ($overall) { Write-Host "`nGAUNTLET EXECUTED SUCCESSFULLY (qualification: $($report.qualification))`n" -ForegroundColor Green; exit 0 }
else { Write-Host "`nGAUNTLET FAILED`n" -ForegroundColor Red; exit 1 }