From 498fbc15e665e06f905096afa6f65f31c300ece0 Mon Sep 17 00:00:00 2001 From: Rostislav Rotaru Date: Fri, 13 Nov 2026 13:26:14 -0800 Subject: [PATCH 1/4] Start the flake proof's display on a display Xvfb picks, behind a cookie Display mode started each run under xvfb-run --auto-servernum, whose check-then-start can give two jobs on one machine the same display. Each run now starts its own Xvfb with -displayfd, as the build and test's end-to-end job does, opens it only to a new xauth cookie, waits at most 30 seconds for the display's number, and stops Xvfb after the run. Agent: DEV1 (developer) Agent-Model: Anthropic Opus 5.5, Medium --- .github/workflows/flake-proof.yml | 33 ++++++++++++++++++++++++++++--- README.md | 2 +- 2 files changed, 31 insertions(+), 4 deletions(-) diff --git a/.github/workflows/flake-proof.yml b/.github/workflows/flake-proof.yml index 27cea7b..01902b9 100644 --- a/.github/workflows/flake-proof.yml +++ b/.github/workflows/flake-proof.yml @@ -266,10 +266,10 @@ jobs: npm run "$script" done - - name: Install the keyring on GitHub's runner + - name: Install the virtual display, the session bus and the keyring on GitHub's runner if: inputs.display && runner.os == 'Linux' && runner.environment == 'github-hosted' timeout-minutes: 5 - run: sudo apt-get update && sudo apt-get install -y --no-install-recommends gnome-keyring + run: sudo apt-get update && sudo apt-get install -y --no-install-recommends xvfb xauth dbus-daemon dbus-bin gnome-keyring - name: Run the tests env: @@ -286,10 +286,15 @@ jobs: read -r -a reports <<< "$REPORTS" command=(npm run "$SCRIPT" -- ${filters[@]+"${filters[@]}"}) keyring=false + xvfb= + trap 'if [ -n "$xvfb" ]; then kill "$xvfb" 2>/dev/null; fi' EXIT if [ "$VIRTUAL_DISPLAY" = true ] && [ "$RUNNER_OS" = Linux ]; then command -v gnome-keyring-daemon >/dev/null || { echo "::error::gnome-keyring isn't installed on this runner; display mode needs it."; exit 1; } + for program in Xvfb xauth dbus-run-session; do + command -v "$program" >/dev/null || { echo "::error::This runner has no $program, which display mode runs each run under."; exit 1; } + done keyring=true - command=(xvfb-run --auto-servernum --server-args='-screen 0 1920x1080x24' dbus-run-session -- sh -c 'echo | XDG_DATA_HOME="$KEYRING/data" gnome-keyring-daemon --unlock --components=secrets --control-directory="$KEYRING/control" >/dev/null && exec "$@"' sh "${command[@]}") + command=(dbus-run-session -- sh -c 'echo | XDG_DATA_HOME="$KEYRING/data" gnome-keyring-daemon --unlock --components=secrets --control-directory="$KEYRING/control" >/dev/null && exec "$@"' sh "${command[@]}") fi mkdir -p _build/flake-proof passed=0 @@ -301,6 +306,23 @@ jobs: mkdir -p "$KEYRING/data/keyrings" mkdir -m 700 "$KEYRING/control" printf login > "$KEYRING/data/keyrings/default" + # Each run gets a new Xvfb on a display it picks itself with -displayfd, so jobs sharing a machine never race for one, + # open only to a new cookie. Xvfb takes every cookie in its -auth file whatever display the entry names, + # so the cookie goes in under :0 before the display is known, and under the display Xvfb picks once it is. + export XAUTHORITY="$KEYRING/xauthority" + cookie=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n') + xauth -q add :0 . "$cookie" + mkfifo "$KEYRING/display" + Xvfb -displayfd 3 -auth "$XAUTHORITY" -screen 0 1920x1080x24 -nolisten tcp 3> "$KEYRING/display" & + xvfb=$! + display= + read -r -t 30 display < "$KEYRING/display" || true + if [ -z "$display" ]; then + echo "::error::Xvfb didn't start for run $run, so the run has no display." + exit 1 + fi + xauth -q add ":$display" . "$cookie" + export DISPLAY=":$display" fi if "${command[@]}" > "$log" 2>&1; then passed=$(( passed + 1 )) @@ -310,6 +332,11 @@ jobs: echo "::error::Run $run failed; its log is kept in the job's artifact." tail -n 60 "$log" fi + if [ -n "$xvfb" ]; then + kill "$xvfb" 2>/dev/null + wait "$xvfb" 2>/dev/null + xvfb= + fi for folder in ${reports[@]+"${reports[@]}"}; do if [ -e "$folder" ]; then mkdir -p "_build/flake-proof/run-$run/$(dirname "$folder")" diff --git a/README.md b/README.md index c0d37fc..5ae8217 100644 --- a/README.md +++ b/README.md @@ -385,7 +385,7 @@ jobs: - A repository proves an end-to-end test with four more inputs, which its caller adds to its own `workflow_dispatch` and passes on: - `script`: the npm script each run runs, such as `test:e2e`. - `prepare`: npm scripts each job runs once, after `npm run build` and before the runs, separated by spaces, such as an app's install and build. They run with the build's token and with no package's install scripts. - - `display`: a boolean, off by default. On Linux it runs each run under `xvfb-run`, on a 1920×1080 screen, and `dbus-run-session`, for a test that opens windows, with an unlocked `gnome-keyring` secret service on a new empty keyring, for a test whose app keeps a secret; it installs `gnome-keyring` on a GitHub-hosted runner and fails on a self-hosted one without it. Windows and macOS runners have a display and a secret store of their own. + - `display`: a boolean, off by default. On Linux it starts a new Xvfb for each run, on a 1920×1080 screen, on a free display Xvfb picks itself with `-displayfd`, so jobs sharing a machine don't race for one, open only to clients holding a new `xauth` cookie, which the run finds through `XAUTHORITY`, and runs the run inside `dbus-run-session`, for a test that opens windows, with an unlocked `gnome-keyring` secret service on a new empty keyring, for a test whose app keeps a secret. It installs `xvfb`, `xauth`, `dbus-daemon`, `dbus-bin` and `gnome-keyring` on a GitHub-hosted runner, fails on a self-hosted one that lacks Xvfb, `xauth`, `dbus-run-session` or `gnome-keyring-daemon`, and fails when Xvfb names no display within 30 seconds. Windows and macOS runners have a display and a secret store of their own. - `reports`: the folders in the checkout the script writes its report and results to, separated by spaces, such as `_build/ui/report _build/ui/results`. After each run, each folder that exists moves into the artifact as `run-/`, so every run keeps its own report beside its log; the plan job fails on a folder outside the checkout, inside `_build/flake-proof`, or holding it, such as `_build`. For one end-to-end spec, the caller passes the spec and its runner's options in `filter`, which go after `--`: with `script` `test:ui`, `filter` `tests/e2e/window.spec.ts:42 --retries 0` runs `npm run test:ui -- tests/e2e/window.spec.ts:42 --retries 0`, so the runner retries nothing and a flaky run fails. From 15c113ef0b6df9e0e004f45e7b93fd524a4035f1 Mon Sep 17 00:00:00 2001 From: Rostislav Rotaru Date: Fri, 13 Nov 2026 13:36:06 -0800 Subject: [PATCH 2/4] Keep a dead Xvfb from ending the flake proof's run step, and create the cookie file first Stopping Xvfb, after a run and in the exit trap, ignores kill's and wait's status, so an Xvfb that has already died doesn't end the step under errexit. The cookie file exists before xauth adds to it, so xauth doesn't print that it doesn't exist. Agent: DEV1 (developer) Agent-Model: Anthropic Opus 5.5, Medium --- .github/workflows/flake-proof.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/flake-proof.yml b/.github/workflows/flake-proof.yml index 01902b9..618d996 100644 --- a/.github/workflows/flake-proof.yml +++ b/.github/workflows/flake-proof.yml @@ -287,7 +287,7 @@ jobs: command=(npm run "$SCRIPT" -- ${filters[@]+"${filters[@]}"}) keyring=false xvfb= - trap 'if [ -n "$xvfb" ]; then kill "$xvfb" 2>/dev/null; fi' EXIT + trap 'if [ -n "$xvfb" ]; then kill "$xvfb" 2>/dev/null || true; fi' EXIT if [ "$VIRTUAL_DISPLAY" = true ] && [ "$RUNNER_OS" = Linux ]; then command -v gnome-keyring-daemon >/dev/null || { echo "::error::gnome-keyring isn't installed on this runner; display mode needs it."; exit 1; } for program in Xvfb xauth dbus-run-session; do @@ -310,6 +310,7 @@ jobs: # open only to a new cookie. Xvfb takes every cookie in its -auth file whatever display the entry names, # so the cookie goes in under :0 before the display is known, and under the display Xvfb picks once it is. export XAUTHORITY="$KEYRING/xauthority" + touch "$XAUTHORITY" cookie=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n') xauth -q add :0 . "$cookie" mkfifo "$KEYRING/display" @@ -333,8 +334,8 @@ jobs: tail -n 60 "$log" fi if [ -n "$xvfb" ]; then - kill "$xvfb" 2>/dev/null - wait "$xvfb" 2>/dev/null + kill "$xvfb" 2>/dev/null || true + wait "$xvfb" 2>/dev/null || true xvfb= fi for folder in ${reports[@]+"${reports[@]}"}; do From 0edc485b4ff5cee6900a09ff016f857271bd0b1c Mon Sep 17 00:00:00 2001 From: Rostislav Rotaru Date: Sat, 14 Nov 2026 11:50:02 -0800 Subject: [PATCH 3/4] Name a flake-proof run that hangs, and give the Xvfb limit's reason Each run may take an even share of the job's time still left, keeping three minutes for the artifact, through GNU coreutils' timeout or gtimeout; a run that reaches it is named with an error and counted as failed, instead of the job's timeout ending the job without naming the run. A runner with neither, such as a stock macOS one, runs without the limit and says so. The README gives the reasons for the per-run limit and for the 30 seconds Xvfb has to name its display. Agent: DEV1 (developer) Agent-Model: Anthropic Opus 5.5, Medium --- .github/workflows/flake-proof.yml | 34 ++++++++++++++++++++++++++++++- README.md | 3 ++- 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/.github/workflows/flake-proof.yml b/.github/workflows/flake-proof.yml index 618d996..10cce4f 100644 --- a/.github/workflows/flake-proof.yml +++ b/.github/workflows/flake-proof.yml @@ -200,6 +200,9 @@ jobs: matrix: include: ${{ fromJSON(needs.plan.outputs.matrix) }} steps: + - name: Note when the job started + run: echo "JOB_STARTED=$(date +%s)" >> "$GITHUB_ENV" + - name: Check out the tested revision uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -280,6 +283,7 @@ jobs: RUNS: ${{ matrix.runs }} TARGET: ${{ matrix.target }} JOB: ${{ matrix.job }} + JOB_MINUTES: ${{ inputs.timeout-minutes }} run: | set -uo pipefail read -r -a filters <<< "$FILTER" @@ -296,6 +300,19 @@ jobs: keyring=true command=(dbus-run-session -- sh -c 'echo | XDG_DATA_HOME="$KEYRING/data" gnome-keyring-daemon --unlock --components=secrets --control-directory="$KEYRING/control" >/dev/null && exec "$@"' sh "${command[@]}") fi + # Each run may take an even share of the job's time still left, keeping three minutes for the artifact, so a run that hangs is named and counted as failed + # before the job's timeout-minutes end the job unnamed; a runner with neither timeout nor gtimeout, such as a stock macOS one, runs without that limit. + limiter= + for candidate in timeout gtimeout; do + # Only GNU coreutils' timeout, not Windows' own timeout.exe, which only waits. + if command -v "$candidate" >/dev/null && "$candidate" --version 2>/dev/null | grep coreutils >/dev/null; then + limiter=$candidate + break + fi + done + if [ -z "$limiter" ]; then + echo "::notice::This runner has neither timeout nor gtimeout, so a run that hangs is ended only by the job's timeout-minutes, without its number." + fi mkdir -p _build/flake-proof passed=0 failed=0 @@ -325,9 +342,24 @@ jobs: xauth -q add ":$display" . "$cookie" export DISPLAY=":$display" fi - if "${command[@]}" > "$log" 2>&1; then + status=0 + if [ -n "$limiter" ]; then + limit=$(( (JOB_MINUTES * 60 - ($(date +%s) - JOB_STARTED) - 180) / (RUNS - run + 1) - 30 )) + if [ "$limit" -lt 60 ]; then + echo "::error::The job's timeout-minutes leave run $run $limit seconds; raise timeout-minutes or lower runs." + exit 1 + fi + "$limiter" --kill-after=30 "$limit" "${command[@]}" > "$log" 2>&1 || status=$? + else + "${command[@]}" > "$log" 2>&1 || status=$? + fi + if [ "$status" = 0 ]; then passed=$(( passed + 1 )) echo "Run $run passed." + elif [ -n "$limiter" ] && { [ "$status" = 124 ] || [ "$status" = 137 ]; }; then + failed=$(( failed + 1 )) + echo "::error::Run $run reached its limit of $limit seconds and was stopped; its log is kept in the job's artifact." + tail -n 60 "$log" else failed=$(( failed + 1 )) echo "::error::Run $run failed; its log is kept in the job's artifact." diff --git a/README.md b/README.md index 5ae8217..07eacba 100644 --- a/README.md +++ b/README.md @@ -385,13 +385,14 @@ jobs: - A repository proves an end-to-end test with four more inputs, which its caller adds to its own `workflow_dispatch` and passes on: - `script`: the npm script each run runs, such as `test:e2e`. - `prepare`: npm scripts each job runs once, after `npm run build` and before the runs, separated by spaces, such as an app's install and build. They run with the build's token and with no package's install scripts. - - `display`: a boolean, off by default. On Linux it starts a new Xvfb for each run, on a 1920×1080 screen, on a free display Xvfb picks itself with `-displayfd`, so jobs sharing a machine don't race for one, open only to clients holding a new `xauth` cookie, which the run finds through `XAUTHORITY`, and runs the run inside `dbus-run-session`, for a test that opens windows, with an unlocked `gnome-keyring` secret service on a new empty keyring, for a test whose app keeps a secret. It installs `xvfb`, `xauth`, `dbus-daemon`, `dbus-bin` and `gnome-keyring` on a GitHub-hosted runner, fails on a self-hosted one that lacks Xvfb, `xauth`, `dbus-run-session` or `gnome-keyring-daemon`, and fails when Xvfb names no display within 30 seconds. Windows and macOS runners have a display and a secret store of their own. + - `display`: a boolean, off by default. On Linux it starts a new Xvfb for each run, on a 1920×1080 screen, on a free display Xvfb picks itself with `-displayfd`, so jobs sharing a machine don't race for one, open only to clients holding a new `xauth` cookie, which the run finds through `XAUTHORITY`, and runs the run inside `dbus-run-session`, for a test that opens windows, with an unlocked `gnome-keyring` secret service on a new empty keyring, for a test whose app keeps a secret. It installs `xvfb`, `xauth`, `dbus-daemon`, `dbus-bin` and `gnome-keyring` on a GitHub-hosted runner, fails on a self-hosted one that lacks Xvfb, `xauth`, `dbus-run-session` or `gnome-keyring-daemon`, and fails when Xvfb names no display within 30 seconds: in the trials Xvfb named its display in under a second, so 30 seconds only guards against a hung Xvfb, with a wide margin. Windows and macOS runners have a display and a secret store of their own. - `reports`: the folders in the checkout the script writes its report and results to, separated by spaces, such as `_build/ui/report _build/ui/results`. After each run, each folder that exists moves into the artifact as `run-/`, so every run keeps its own report beside its log; the plan job fails on a folder outside the checkout, inside `_build/flake-proof`, or holding it, such as `_build`. For one end-to-end spec, the caller passes the spec and its runner's options in `filter`, which go after `--`: with `script` `test:ui`, `filter` `tests/e2e/window.spec.ts:42 --retries 0` runs `npm run test:ui -- tests/e2e/window.spec.ts:42 --retries 0`, so the runner retries nothing and a flaky run fails. Before it installs anything, each job checks that `script` and every `prepare` script are npm scripts in the package's `package.json`, and fails otherwise. - A test job may take `timeout-minutes`, 120 by default and up to 360, which a caller with a slow suite raises. + Each run may take an even share of the job's time still left, keeping three minutes for the artifact, so a run that hangs is stopped, named with `::error` as "Run reached its limit of seconds" and counted as failed before the job's timeout ends the job without naming it; the job fails at once when a run's share is under a minute. The limit runs through GNU coreutils' `timeout`, or `gtimeout`; on a runner with neither, such as a stock macOS one, runs have no limit of their own and the job says so with a notice. - The jobs install from the registries and run on the runners [Shared settings](#shared-settings) names; a private repository's jobs read the organization's private packages from GitHub's registry with the run's own token, which `packages: read` allows, and a public repository's jobs hand npm no token. - The caller grants `contents: read` and `packages: read`, since a called job gets no permission its caller didn't grant. From 03bc6f18e2e5291871fba137c3550e425181c01f Mon Sep 17 00:00:00 2001 From: Rostislav Rotaru Date: Sat, 14 Nov 2026 11:58:18 -0800 Subject: [PATCH 4/4] Bound every flake-proof run with a watchdog in the shell, on every runner A runner without GNU timeout, such as a stock macOS one, ran its runs without a limit. Each run now starts in the background with a watchdog that sends TERM at the run's limit and KILL 30 seconds later, and is stopped once the run ends; a run the watchdog ended gets the same error and counts as failed. Agent: DEV1 (developer) Agent-Model: Anthropic Opus 5.5, Medium --- .github/workflows/flake-proof.yml | 50 ++++++++++++++++--------------- README.md | 2 +- 2 files changed, 27 insertions(+), 25 deletions(-) diff --git a/.github/workflows/flake-proof.yml b/.github/workflows/flake-proof.yml index 10cce4f..79864b8 100644 --- a/.github/workflows/flake-proof.yml +++ b/.github/workflows/flake-proof.yml @@ -301,18 +301,7 @@ jobs: command=(dbus-run-session -- sh -c 'echo | XDG_DATA_HOME="$KEYRING/data" gnome-keyring-daemon --unlock --components=secrets --control-directory="$KEYRING/control" >/dev/null && exec "$@"' sh "${command[@]}") fi # Each run may take an even share of the job's time still left, keeping three minutes for the artifact, so a run that hangs is named and counted as failed - # before the job's timeout-minutes end the job unnamed; a runner with neither timeout nor gtimeout, such as a stock macOS one, runs without that limit. - limiter= - for candidate in timeout gtimeout; do - # Only GNU coreutils' timeout, not Windows' own timeout.exe, which only waits. - if command -v "$candidate" >/dev/null && "$candidate" --version 2>/dev/null | grep coreutils >/dev/null; then - limiter=$candidate - break - fi - done - if [ -z "$limiter" ]; then - echo "::notice::This runner has neither timeout nor gtimeout, so a run that hangs is ended only by the job's timeout-minutes, without its number." - fi + # before the job's timeout-minutes end the job unnamed. A watchdog in this shell bounds it, on every runner, since not every runner has GNU timeout. mkdir -p _build/flake-proof passed=0 failed=0 @@ -342,21 +331,34 @@ jobs: xauth -q add ":$display" . "$cookie" export DISPLAY=":$display" fi - status=0 - if [ -n "$limiter" ]; then - limit=$(( (JOB_MINUTES * 60 - ($(date +%s) - JOB_STARTED) - 180) / (RUNS - run + 1) - 30 )) - if [ "$limit" -lt 60 ]; then - echo "::error::The job's timeout-minutes leave run $run $limit seconds; raise timeout-minutes or lower runs." - exit 1 - fi - "$limiter" --kill-after=30 "$limit" "${command[@]}" > "$log" 2>&1 || status=$? - else - "${command[@]}" > "$log" 2>&1 || status=$? + limit=$(( (JOB_MINUTES * 60 - ($(date +%s) - JOB_STARTED) - 180) / (RUNS - run + 1) - 30 )) + if [ "$limit" -lt 60 ]; then + echo "::error::The job's timeout-minutes leave run $run $limit seconds; raise timeout-minutes or lower runs." + exit 1 fi - if [ "$status" = 0 ]; then + reached="$RUNNER_TEMP/flake-proof-limit-$JOB-$run" + status=0 + "${command[@]}" > "$log" 2>&1 & + pid=$! + # The watchdog sends TERM at the limit and KILL 30 seconds later; its own sleeps go when it is stopped, so nothing outlives the run. + ( + trap 'kill "$nap" 2>/dev/null; exit 0' TERM + sleep "$limit" & nap=$! + wait "$nap" + touch "$reached" + kill -TERM "$pid" 2>/dev/null + sleep 30 & nap=$! + wait "$nap" + kill -KILL "$pid" 2>/dev/null + ) > /dev/null 2>&1 & + watchdog=$! + wait "$pid" || status=$? + kill -TERM "$watchdog" 2>/dev/null || true + wait "$watchdog" 2>/dev/null || true + if [ "$status" = 0 ] && [ ! -e "$reached" ]; then passed=$(( passed + 1 )) echo "Run $run passed." - elif [ -n "$limiter" ] && { [ "$status" = 124 ] || [ "$status" = 137 ]; }; then + elif [ -e "$reached" ]; then failed=$(( failed + 1 )) echo "::error::Run $run reached its limit of $limit seconds and was stopped; its log is kept in the job's artifact." tail -n 60 "$log" diff --git a/README.md b/README.md index 07eacba..3184eea 100644 --- a/README.md +++ b/README.md @@ -392,7 +392,7 @@ jobs: Before it installs anything, each job checks that `script` and every `prepare` script are npm scripts in the package's `package.json`, and fails otherwise. - A test job may take `timeout-minutes`, 120 by default and up to 360, which a caller with a slow suite raises. - Each run may take an even share of the job's time still left, keeping three minutes for the artifact, so a run that hangs is stopped, named with `::error` as "Run reached its limit of seconds" and counted as failed before the job's timeout ends the job without naming it; the job fails at once when a run's share is under a minute. The limit runs through GNU coreutils' `timeout`, or `gtimeout`; on a runner with neither, such as a stock macOS one, runs have no limit of their own and the job says so with a notice. + Each run may take an even share of the job's time still left, keeping three minutes for the artifact, so a run that hangs is stopped, named with `::error` as "Run reached its limit of seconds" and counted as failed before the job's timeout ends the job without naming it; the job fails at once when a run's share is under a minute. A watchdog in the job's own shell bounds each run on every runner, sending `TERM` at the limit and `KILL` 30 seconds later, since not every runner has GNU coreutils' `timeout`. - The jobs install from the registries and run on the runners [Shared settings](#shared-settings) names; a private repository's jobs read the organization's private packages from GitHub's registry with the run's own token, which `packages: read` allows, and a public repository's jobs hand npm no token. - The caller grants `contents: read` and `packages: read`, since a called job gets no permission its caller didn't grant.