test: eliminate intermittent macOS gtest-discovery build failure - #556
Conversation
michalhosna
left a comment
There was a problem hiding this comment.
@michalhosna made 1 comment.
Reviewable status: 0 of 1 files reviewed, 1 unresolved discussion (waiting on gmarzot).
-- commits line 14 at r1:
Homebrew now ships 4.4.2 (See https://formulae.brew.sh/formula/cmake).
So, is this still a problem?
afrind
left a comment
There was a problem hiding this comment.
I'm ok with this, but maybe it's not actually going to fix it?
@afrind reviewed 1 file and all commit messages, and made 2 comments.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on michalhosna).
Previously, michalhosna (Michal Hošna) wrote…
Homebrew now ships 4.4.2 (See https://formulae.brew.sh/formula/cmake).
So, is this still a problem?
What's on our CI runner?
I am reading now that this problem will fade out in a few weeks. the runners are mixed and the bad version can still crop up. the restructure is slightly better even after 4.4.2 roles out... conserves runner cycles.. but doesnt really translate into any observable build perf benefit ~1sec we can close or merge.. you guys decide? eh i see a check mark.. will merge |
|
we should not see it again w/ this change and some minor optimization bonus |
gmarzot
left a comment
There was a problem hiding this comment.
@gmarzot made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on afrind and michalhosna).
CMake 4.4.0's generated discovery scripts omit TEST_TARGET, so every target's POST_BUILD discovery shares one JSON file (cmake_test_discovery_e3b0c44298.json, sha256 of the empty string). Ninja links test binaries in parallel and their discovery steps race on that file: one truncates it while another parses, producing the intermittent macOS failure ParseTestList.cmake:96: string sub-command JSON failed parsing json string ... Line 1, Column 1: Syntax error Fixed upstream in cmake 4.4.1, but Homebrew CI runners ship 4.4.0. PRE_TEST discovery runs serially at ctest startup, immune to the race on any cmake version. Measured cost: 0.8s to enumerate all 772 tests, cached thereafter (0.12s); build step drops its ~28 post-link discovery executions.
6c89931 to
1619dbb
Compare
gmarzot
left a comment
There was a problem hiding this comment.
@gmarzot resolved 1 discussion.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on gmarzot).
Problem
The macos job intermittently fails at build time with:
(e.g. this run on #533 — passed on rerun, unrelated to the PR's changes.)
Root cause
CMake 4.4.0 (current Homebrew) has a regression in the GoogleTest module refactor: the generated
*_discovery.cmakescripts omitTEST_TARGET, so the per-target hash that keys the discovery JSON file is computed from the empty string. Every test target inbuild/testtherefore shares one file,cmake_test_discovery_e3b0c44298.json(sha256("")), defeating the race protection that hash exists for.In default POST_BUILD mode, discovery runs right after each test binary links — and Ninja links many test binaries near-simultaneously (the failing run linked three within 0.2 s). Concurrent discoveries then race on the shared file: one truncates/removes it while another parses → empty JSON → the error above. Intermittent by nature; the losing target varies. Fixed upstream in cmake 4.4.1, which Homebrew doesn't ship yet.
Fix
Switch discovery to
PRE_TESTmode (oneset()before thegtest_discover_tests()calls). Discovery then runs serially at ctest startup — immune to the race on any cmake version.Cost
ctestinvocation; cached via anIS_NEWER_THANguard thereafter (measured 0.12 s), re-discovering only binaries that actually relinked.Validation (local, linux, cmake 4.2.3)
*_include.cmakefiles now carry the guardedgtest_discover_tests_implcall.ctest -N: all 772 tests enumerated.UpstreamProviderTestnetwork tests fail/hang on this dev box with or without the change — pre-existing local-env issue; CI is the gate for those.)Note for #519:
moqx_add_gtest()there still uses default POST_BUILD discovery, so this line should survive the rework (or move into the helper asDISCOVERY_MODE PRE_TEST).This change is