Skip to content

CI: preserve upstream tests and bound automatic native validation - #9

Draft
jmonster wants to merge 21 commits into
masterfrom
fix/lean-ci-budget
Draft

jmonster wants to merge 21 commits into
masterfrom
fix/lean-ci-budget

Conversation

@jmonster

@jmonster jmonster commented Sep 20, 2026

Copy link
Copy Markdown
Owner

Status: native acceptance is not complete

Current head: b51491b885853316a6fdb64d834fe6f9c59045e3.

Automatic PR run 35649036304 validates this revision. Keep this PR in draft until every selected job succeeds within its existing automatic budget. Earlier revisions, cancelled jobs, unexpectedly skipped checks, and host fixtures are not native acceptance evidence. Warm-cache success does not excuse a reproducible cold-build timeout.

What this PR preserves

Dolphin's Google Test, CMake/CTest, tests, and unittests remain the baseline. Focused tests are included in ordinary POSIX ENABLE_TESTS builds with the backend ON or OFF, without a separate opt-in. Focused checks run on every PR; the fail-closed changed-input selector requires real SDK, rumble, in-process SDL, and affected native validation. Shared production/build/resource changes select both Mac architectures, Linux enabled/disabled, and Windows.

Native jobs retain full application compilation/linking, architecture and dependency checks, user-profile safeguards, packaging, relocation, and exact-archive GUI launch/normal quit/relaunch. Disabled Linux remains a separate full build in a Swift-free Ubuntu image with 512 MiB shared memory and the complete upstream unit suite. ASan/UBSan and compiler-capacity probes remain required. Switch2Kit checks rejects failures, cancellations, and unexpected skips.

Automatic native jobs retain ten-minute setup/build/package/launch budgets; focused CI retains three minutes and twenty-second CTest-command limits; SDK CI retains five minutes. No larger runners, added shards, optional tests, fabricated configure answers, or manual-only replacement coverage.

Build corrections

  • Isolate compiler caching from real configure/link probes and nested SDK/sanitizer configurations; use target-only launchers and preserve Dolphin's manual Windows PCH ordering.
  • Use per-target POSIX PCHs and the shader compiler's own upstream PCH. Reuse upstream test PCHs only for equivalent settings. Preserve separate translation units, generated sources, and custom-command dependencies.
  • Finalize automatic compiler options after target initialization: Windows /Od /Ob0, Apple -O0 including late-enabled Objective-C languages, and Linux -O1. Preserve definitions, sanitizer instrumentation, line-level debug information, and explicit Release/developer defaults.
  • Use the existing Windows runner's LLD for real executable links. Omit PDB/manifest generation only for feature probes, not application links. Give DolphinQt its own PCH with its actual RTTI/Qt settings rather than the core's shared PCH.
  • Remove unnecessary Qt autogen waits for compiled libraries while retaining explicit staging dependencies, existing autogen dependencies, and the generated achievement header.
  • Verify already-fetched native and recursive gitlinks instead of refetching them. Preserve dependency pins and developer checkout behavior.
  • Resolve Xcode compilers together with SDKROOT, retain the real C++23 compile/link/run probe, and avoid incidental Homebrew upgrades of unrelated Qt components.
  • Save bounded compiler-object caches and native diagnostics after failed/cancelled jobs. Cache neither applications, build trees, configure answers, nor test results; keep failing job results red. Build Linux-disabled application/test objects together, then execute the unchanged complete unittests target.

Evidence and outstanding acceptance

The four native execution regressions pass locally with GCC and Clang in approximately 11–12 seconds. These include real compiler/link probes, PCH/generated-source builds, repeated CTest execution, Objective-C/C++ sanitizer-backed execution, and recursive Git pin checks. The added Qt wiring assertions fail against the preceding helper. Native Windows Qt/MSVC and macOS behavior still require the platform jobs.

Run 35643722112 on 705c580 passed focused checks, SDK checks, Linux-enabled qualification, and Apple Silicon qualification; Windows, Intel macOS, and Linux-disabled timed out. Run 35646179278 on b3b17ae passed focused checks, SDK checks, and Apple Silicon qualification; Windows, Intel macOS, and Linux-enabled timed out, and Linux-disabled was skipped. The final gate correctly failed. That run verified that timeout diagnostics and compiler objects are now retained.

The latest correction targets the observed Windows/Linux Qt build tail. Intel macOS configuration profiling is included to identify its separate cold-configure bottleneck. No all-green native result is claimed yet.

Automatic smoke builds are not optimized-release performance qualification. Native archive launch is not Bluetooth hardware/gameplay qualification. Controller behavior, SDK pins, user-data safeguards, and repository protection settings have not been changed. Require Switch2Kit checks before merging.

AI assistance disclosure: prepared with AI assistance; human review required.

Remove full macOS, Linux and Windows application builds, SDK qualification,
and source archives from automatic PR checks. Keep the existing small
production-code regressions and sanitizers in one standard Linux job.

Enforce a shared 60-second suite budget, 20 seconds per command, and a
3-minute job ceiling. Add AGENTS.md and structural workflow policy tests
that reject accidental fan-out, recursive checkout, automatic qualification,
soft-fail gates and raised budgets. Preserve native packaging/launch steps
behind explicit single-target manual dispatches.

AI-assisted change. Policy/budget regression tests passed locally; full
integration verification must use the exact-head fast GitHub Actions run.
No expensive native builds or hardware checks were launched.
GitHub does not allow the runner context in job-level env. Scope the
isolated parser path to its two consuming steps instead, and cover the
invalid job-level configuration in the policy mutation tests.

Local policy and all 15 enforcement tests pass. Keep all runtime budgets,
production regression commands and manual qualification gates unchanged.
Keep every consent/startup assertion intact. The CI-retention check now
checks the actual runner argument tuples, including sanitizer flags,
instead of requiring individual commands inline in the workflow YAML.
The focused retention test and all 15 policy tests pass locally.
@jmonster
jmonster marked this pull request as draft September 20, 2026 20:41
Replace the fork-wide YAML policy and bespoke Python deadline runner with
ordinary CTest registration for the existing focused regression harnesses.
Preserve upstream tests/unittests and run its Google Test suite in the real
Swift-free Linux application build tree.

Restore automatic real SDK/rumble/SDL tests for changed SDK inputs and all
native build, architecture, dependency and exact-package launch checks for
changed production/build inputs. Scope platform-only changes to their own
jobs; test selection against renames, deletions and Git failures. Aggregate
selected job results so the small wiring check cannot hide missing native
qualification. Cancel obsolete runs and avoid duplicate PR triggers.

Use standard CMake sccache launchers and a pinned Swift test-build cache;
never cache a passing test result. Keep application archives for launch
validation but upload successful distributions only on explicit dispatch.
Rewrite AGENTS.md around upstream compatibility, efficient coverage and
honest reporting instead of a blanket ban on native tests.

AI-assisted change. Nine change-selection regressions, CTest enumeration,
the CI-retention regression, Python compilation and YAML parsing passed
locally in a partial checkout. Full exact-head CI must pass; no native
performance improvement or native validation result is assumed.
@jmonster jmonster changed the title CI: one fast PR job, hard runtime budgets, and no automatic native builds CI: preserve upstream CTest, scope native validation, and cache compilation Sep 20, 2026

Copy link
Copy Markdown
Owner Author

Exact-head validation update for e433603bac39fd5d5a1457033dacb481112242f8 (run 35536963207):

  • All eight focused CTest groups passed in 23.02 seconds. This is the focused test time only.
  • The real SDK job passed, including Swift tests, rumble tests and real in-process SDL integration. Its test command block is unchanged from master; tests were not skipped on a cache hit.
  • The selector requested every validation category for this orchestration change. The two native Mac architectures, Linux enabled and Windows jobs have started automatically. Linux's Swift-free disabled build and ordinary upstream unit suite follow the enabled Linux job.
  • Full native compilation/package-launch validation is still in progress. The aggregate Switch2Kit checks must pass; the PR remains draft. No native speedup or equivalent end-to-end seconds-only result is claimed.

The former one-job/manual-only design is superseded. AGENTS.md now requires preserving upstream tests, selecting necessary native validation, avoiding duplicate work and reporting measured cold/warm results without substituting fixture passes for platform qualification.

Remove ENABLE_SWITCH2KIT from test registration. The existing ENABLE_TESTS
path now includes the POSIX fixture regressions even when the application
backend is disabled; preserve the existing Windows harness limitation.
No new test option, target, workflow or backend default is introduced.

Extend the existing retention tests to compare CTest's actual commands in
the standalone and normal parent registrations with the backend OFF/ON.
The regression fails with the old gate and passes with the new wiring.
Supply Clang in the Swift-free Linux test image for the retained compiler
capacity probe. Keep upstream tests, sanitizers and native qualification.

Update AGENTS.md and the CI documentation to prohibit a separate opt-in.
AI-assisted change. Registration regression, Python compilation, YAML
comparison and whitespace checks passed locally in a partial snapshot.
Full exact-head CI still must pass; no native build result is inferred.
The disabled application builds, but upstream PageTableHostMappingTest
crashes with SIGBUS during memory.Init(). The container's default 64 MiB
/dev/shm is smaller than Dolphin's default shared-memory allocation.
All eight Switch2Kit CTest groups passed in the same failed run.

Give only the Swift-free unit-test container 512 MiB of shared-memory
capacity. Check for at least 128 MiB free before installing dependencies
or compiling so this configuration regression fails immediately.

Keep every build, upstream/addition test, sanitizer, dependency/launch
assertion, timeout, and aggregate gate unchanged. No privileged container,
new job, test filter, or application behavior change.

AI-assisted change. Original blob hash verified; YAML and shell syntax
checked; exact preflight accepts adequate capacity and rejects 64 MiB,
below-threshold, malformed/missing capacity and df failure locally.
The actual upstream unit suite still requires exact-head CI verification.

Copy link
Copy Markdown
Owner Author

Linux failure diagnosed; container-only correction pushed

The failure in run 35539103585, linux / disabled is not a compile failure or a failing Switch2Kit assertion. The downloaded Dolphin-without-Switch2Kit-linux-diagnostics artifact shows the application built, all eight Switch2Kit CTest groups passed, and upstream's tests executable terminated with Bus error while entering PageTableHostMappingTest (before its first test case).

That suite calls memory.Init(). Dolphin's Unix memory arena uses shm_open/mmap, and the default GameCube RAM + fake VMEM + L1 allocation exceeds the container's default 64 MiB /dev/shm. I had added upstream unittests without provisioning that container resource. The aggregate check correctly rejected the failed upstream suite.

b23e1ccd69c4855f400fc6936d4b9db11cb6cce0 changes only .github/workflows/switch2kit-linux.yml:

  • Set --shm-size=512m on the Swift-free unit-test container. The image, runner, timeouts, isolation, and compiler caching are unchanged; no privileged/host-IPC workaround.
  • Check for at least 128 MiB free /dev/shm before installing dependencies or compiling. Removing the resource setting will now fail immediately with an explicit message, rather than after a native build.

Every existing build/test/dependency/launch command and the aggregate gate are unchanged. No test filtering, skipping, reduced assertions, disabled sanitizers, added options, or production-code changes.

Local validation: original/pushed Git blob hashes verified; YAML and all shell blocks parse; executing the exact new preflight accepts 128/512 MiB and rejects 64 MiB, below-threshold, invalid/empty values, and df failure. These checks are not a substitute for the real unit suite.

Current-head run 35543671612 is in progress. This PR remains draft while the real upstream suite and native jobs validate the correction.

@jmonster
jmonster marked this pull request as ready for review September 21, 2026 01:19
@jmonster
jmonster marked this pull request as draft September 21, 2026 01:46
Keep complete native application builds and all existing SDK, sanitizer,
upstream unit, architecture, dependency, relocation and exact-archive
launch assertions. Do not replace native checks with fixtures or manual
labels. Automatic native jobs now have a 10-minute ceiling including setup.
A timeout remains an aggregate failure, not a performance success.

For automatic macOS/Windows smoke builds, keep Release defines and CRT/ABI
but disable expensive C/C++ optimization; these jobs do not benchmark the
emulator. Linux optimized builds and Swift SDK configurations are unchanged.
Explicit artifact builds and local defaults retain optimized Release flags.
This does not claim optimized macOS/Windows code-generation qualification.

Cache the pinned Windows Swift installer; avoid upgrading installed Homebrew
build tools; use the standard runner's existing logical CPUs; expose compiler
cache statistics. No new jobs, runners, matrices, triggers or test opt-ins.

AGENTS.md now requires measured exact-head jobs within budget, including cold
cache runs, and forbids deadline inflation or reporting timed-out work as fast.

AI-assisted. Original Git blobs verified. YAML and POSIX shell syntax,
retained validation-step equality, automatic/manual argument construction,
a real CMake compiler-flag fixture and diff whitespace checks passed locally.
Native performance and PowerShell execution still require exact-head CI.
@jmonster jmonster changed the title CI: preserve upstream CTest, scope native validation, and cache compilation CI: preserve upstream tests and bound automatic native validation Sep 21, 2026
…ffic

Apply smoke flags after upstream MSVC initialization through CMake's project
include hook; keep upstream build sources and local defaults unchanged.
Reuse the existing upstream PCH header per POSIX target and preserve Windows'
existing shared PCH. Keep every translation unit, native platform, sanitizer,
SDK test, upstream unit test and exact-archive launch assertion.

Restore bounded local compiler caches once per job instead of issuing remote
cache requests for each compilation. Use the same distro Clang 18, build path
and install prefix for the isolated optimized Linux ON/OFF configurations so
compatible objects can be shared. Do not restore built applications or tests.
Fetch pinned submodules in parallel and install only required Qt components.

Extend the existing CTest-registered CI regressions to catch MSVC flag
shadowing and validate real mixed C/C++ per-target precompiled-header builds.
Keep the 10-minute automatic job budgets and all native validation steps.

AI-assisted. Verified original blob hashes, YAML/shell syntax, unchanged
native job topology/budgets and validation-step bodies. All 12 CI setup and
selection tests passed locally; a shadowed-MSVC-flags mutation failed as
expected. Native performance remains subject to the exact-head Actions run.
Compile files with source-specific flags normally instead of forcing them
to consume a target PCH compiled with different ISA/language/define options.
In particular, upstream ARM crypto files retain their architecture flags.
No source, test, sanitizer or native check is excluded from compilation.

Extend the existing Clang fixture with a subdirectory-scoped C++20 source
beside a C++23 target PCH. The fixture fails without this correction and
passes with it, while checking the special file still compiles with its
own options and the ordinary files use the PCH. All 12 existing and new CI
setup/selection tests pass locally. Native timings still require CI.

AI-assisted.
Homebrew intentionally installs split Qt formulae with HOMEBREW_PREFIX as
the Qt installation root. Pointing Qt6_DIR at the qtbase keg made Qt's
component lookup miss the separately installed qtsvg keg. Use the common
linked prefix and verify both package configs before configuring Dolphin.
Retain the minimal qtbase/qtsvg dependency set, every native check and the
ten-minute automatic job limit.

AI-assisted.

Copy link
Copy Markdown
Owner Author

Pushed e511b0d5e1bd6699b93f45c59476b4a2017a9033 on this PR, preserving the intervening per-source PCH fix 01e1fea.

The last macOS failure was Qt package discovery, not missing Svg installation: qtbase and qtsvg 6.11.2 were both installed, but Qt was rooted at the qtbase keg. Homebrew's split-module formula explicitly uses the shared Homebrew prefix as Qt's installation root. The workflow now uses that shared prefix and an explicit Qt6_DIR, and checks the two linked package configs immediately after dependency installation. No full Qt metapackage, skipped component, changed source, reduced test, or increased timeout.

Current-head verification: https://github.com/jmonster/dolphin/actions/runs/35555759248 . This is not yet acceptance evidence; every selected native build/launch and the upstream unit suite must actually pass inside the automatic budgets before this PR is marked ready.

Clang 18 with Noble's libstdc++ does not expose std::expected, so the
native build failed in DiscIO after compiling hundreds of objects. Use
Ubuntu's packaged Clang 19 for both application configurations, with a
tiny compiler/library probe before any application build. Keep the
ordinary Clang/GCC regression probes, optimized builds, Swift-free
disabled image, full upstream tests, native launch checks and budgets.

AI-assisted.
Fetch complete pinned submodule updates concurrently, verify every selected
HEAD against the committed gitlink, and exclude only foreign Windows binaries
and Android-only dependencies. Keep the actual native build graph and tests.

Do not wrap configure probes or native PCH consumers in sccache. Retain the
bounded compiler-object cache for ordinary targets. Apply the MSVC smoke
options as inherited directory options so child project flag initialization
cannot silently restore /O2. Upstream and explicit release defaults stay intact.

Extend the existing CTest CI group with real local-git pin regressions and
CMake cache/PCH/child-scope checks. No raised budgets, skipped assertions,
manual-only replacement, runner upgrades or build shards.

AI-assisted.
Keep native configure and nested SDK/sanitizer probes outside compiler-object
caches, explicitly clear inherited PCH launchers, and recognize upstream's
manual MSVC build_pch/use_pch implementation. Preserve its ordering edges.

Reduce automatic Linux compile cost with optimized -O1 builds and POSIX
line-level debug information. Extend per-target upstream PCH use and let
CMake optimize redundant static/object-library ordering without removing
translation units or generated-source dependencies. Manual artifacts retain
upstream Release optimization and debug defaults.

Verify already-fetched recursive gitlinks in CI helpers rather than repeating
unfiltered updates. Avoid unrelated Homebrew dependent upgrades and repeated
Homebrew/toolchain-selector processes. Use a container-visible Linux object
cache path and a fresh cache namespace for cold validation.

Add executable CMake/cache/PCH/generated-source/CTest and recursive local-Git
regressions to ordinary CTest. Existing upstream tests, SDK regressions,
sanitizers, native archive launch checks, runners and job budgets stay intact.

AI-assisted; native acceptance must be established on this exact PR head.
Retain the PCH/Release assertions with the automatic Linux -O1 preset.
Keep standalone scope fixtures independent of the native job's private
compiler-cache environment. All 16 change-selection/setup tests pass.

AI-assisted.
Use the existing runner's LLD for automatic Windows links and real Release
configure probes, while retaining MSVC compilation, the upstream manual PCH,
all feature/link checks, and explicit artifact defaults. Compile glslang's
own pinned parser PCH rather than repeatedly parsing it for every source.
Restore normal upstream dependency ordering instead of the unproven eager
static-library scheduling optimization.

Resolve and validate SDKROOT together with direct Xcode compiler paths so
standalone sanitizer builds retain the complete macOS SDK. Add a real
C++23 atomic compile/link/run toolchain probe before native configuration.
Report active Ninja workers and elapsed build time on the same runners.

Existing local setup and executable CMake/Git regressions pass with GCC,
Clang, and sanitizer-enabled fixtures. Full native acceptance remains a
requirement of this exact PR head; no budgets or coverage were reduced.

AI-assisted; human review required.
Apply automatic MSVC Release options after target initialization, grouping
the final /Od /Ob0 pair so CMake option deduplication cannot discard it.
Keep explicit Release artifacts and all upstream runtime/debug flags intact.

Use CMake's dependency optimizer for automatic static/object targets,
retaining explicit dependencies, generated sources, custom-command side
effects and final link inputs. Extend per-target PCH to upstream test
object libraries without changing translation units or selected tests.

Extend the existing executable CMake regressions to cover final compiler
option order, explicit artifact defaults, native test-library PCH builds,
and the optimized Ninja compile-order edge. GCC and Clang fixture runs
both pass; full native/platform acceptance remains required in CI.

No job, runner, time budget, SDK pin, production controller behavior,
sanitizer check, upstream unit test or archive-launch requirement changes.

AI-assisted; human review required.
Avoid generating one PCH for every one-source upstream test target. Reuse
an object-test producer only when target settings, directory compiler flags
and dependency usage requirements match; isolate differing definitions.
Never reuse the final test executable, which would introduce a cycle.

Keep the early MSVC child-directory options expected by the setup hook,
while the final grouped target options remain authoritative. This fixes
the setup-contract failure reported by automatic run 35637581972 without
removing or weakening that regression.

Existing GCC and Clang executable regressions pass, including shared-PCH
reuse and isolation of differing compile definitions. Native acceptance
and the complete upstream suite remain mandatory on the unchanged jobs.

AI-assisted; human review required.
Finalize Release target options after Objective-C and Objective-C++ are
enabled, so automatic native builds do not silently retain -O3 for those
sources. Preserve explicit Release artifacts, Debug settings, definitions,
sanitizer instrumentation, and all existing native acceptance checks.

Add a real Clang Objective-C/C++ compile, sanitizer link, and CTest execution
regression across automatic Release, explicit Release, and Debug in one
reconfigured build tree. All four native execution regressions pass locally
with GCC and Clang; native platform acceptance is still required.

AI-assisted; human review required.
…compilation

Split compiler-object restore/save using the existing pinned cache action.
Save valid objects even after failed or cancelled native validation, under
unique run/attempt keys so reruns can advance an immutable cache. Never
cache build trees, applications, configure answers or test results. Preserve
all job results and ten-minute automatic acceptance budgets. Collect native
diagnostics on cancellation as well as failure.

Build the disabled Linux application and complete upstream tests executable
in one Ninja graph, then execute the unchanged mandatory unittests target.
This avoids a separate serial tail of test-object compilation without
removing any source, test, sanitizer, dependency or launch assertion.

Parsed before/after workflow validation confirms unchanged runners, budgets,
environments and acceptance commands except the combined Linux build.
Native acceptance remains required on this exact revision.

AI-assisted; human review required.
…aders

Give DolphinQt its own MSVC PCH with its existing RTTI and Qt definitions,
without changing the core's manual shared PCH. Schedule Qt code generation
without waiting for compiled libraries, retaining explicit staging targets,
existing autogen dependencies and the generated achievement header.

Keep feature probes as real executable links while omitting only their
unneeded Windows PDB and manifest work; application link settings remain
unchanged. Record CMake profiling and build-graph diagnostics for the
remaining Intel macOS configure-time bottleneck.

Extend the existing native wiring regression for Qt PCH/cache isolation,
generated-header and explicit dependency preservation, and manual artifact
settings. All four local GCC/Clang regressions pass (11-12 seconds); the
prior helper fails the added regression. Native acceptance remains required.
No runners, budgets, translation units, tests, SDK pins or launch checks
were removed or relaxed.

AI-assisted; human review required.
Apply the runner-images PerfPowerServices workaround on the disposable Intel
macOS runner: the retained process profile showed it consuming 98.5% CPU.
Keep Apple Silicon and developer machines untouched.

Build the real Windows SDK and complete application in one automatic Ninja
graph instead of leaving native compilation idle during the 79-second SDK
phase. Preserve SDK dependency ordering, fatal errors, a final incremental
application check, and the developer/explicit-artifact SDK-first path.
Submit Qt autogen and the expensive Linux emitter test early within the same
complete graph, then execute the unchanged upstream unittests target.
Use Ninja's CPU+2 queue headroom on the existing standard runners; do not add
CPUs, paid runners, shards, jobs, skipped sources or reduced test coverage.

Select the installed native LLVM 19 ELF linker for automatic Linux Clang
application/test links and real executable probes through a private CI-only
setting. Preserve explicit artifact, GCC and nested SDK defaults. Positive
compile/link/run and missing-symbol negative controls pass for automatic and
explicit modes. Existing four native execution regressions pass with GCC
and Clang, including sanitizer, generated-source/PCH and recursive-pin checks.

Start v4 compiler-object caches to measure the initial native run cold rather
than hide a cold failure behind prior objects. All ten-minute automatic job
budgets, optimized Linux flags, mandatory SDK/sanitizer/upstream/GUI archive
checks, dependency pins, controller behavior and user-data safeguards remain.
Exact-head native acceptance is still required.

AI-assisted; human review required.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant