run-tests.php retries any test it considers flaky and prints only the second
attempt's verdict, with (warn: Test passed on retry attempt) as the whole
record. A test is flaky by its judgement when its --FILE-- section calls
disk_free_space, hrtime, microtime, sleep or usleep (is_flaky,
run-tests.php:3118, reached through error_may_be_retried). The first
attempt's output is discarded, so a defect that lands on one run in two is
invisible in suite output.
337 of our 497 phpt files call one of those in their FILE section. Counted
by extracting the FILE section of each file under tests/phpt and matching the
five names — the same test is_flaky applies.
This is not hypothetical. websocket/035-recv-queue-overflow read
client saw close: NULL on 5 of 10 runs while the suite reported it green, and
the defect behind it (#305) survived a local run of 0 failures and a full CI
Windows job. It was found only by running the test outside run-tests.
Live today: tls/003-tls-handshake reports passed on retry attempt on the
Windows job of c5a5379. What its first attempt saw is gone.
Proposed
Two halves, and the first is cheap:
- Take the clock out of the tests that do not need it.
usleep(20000) is
delay(20) in a coroutine context; a microtime() deadline is a bounded
attempt count. 035 shows both substitutions. A test that keeps the retry
should say why in a comment, so the exemption is a decision rather than an
accident.
- Make the retry visible. A warned test is a test whose verdict came from
the second attempt; the first attempt's output belongs in the artifact, and
the Windows job should probably treat a warn as a failure once the count is
low enough to allow it.
Order matters: the count has to come down before the gate can go up, or every
job goes red on the same 337 files.
Found while fixing #305.
run-tests.phpretries any test it considers flaky and prints only the secondattempt's verdict, with
(warn: Test passed on retry attempt)as the wholerecord. A test is flaky by its judgement when its
--FILE--section callsdisk_free_space,hrtime,microtime,sleeporusleep(is_flaky,run-tests.php:3118, reached througherror_may_be_retried). The firstattempt's output is discarded, so a defect that lands on one run in two is
invisible in suite output.
337 of our 497 phpt files call one of those in their FILE section. Counted
by extracting the FILE section of each file under
tests/phptand matching thefive names — the same test
is_flakyapplies.This is not hypothetical.
websocket/035-recv-queue-overflowreadclient saw close: NULLon 5 of 10 runs while the suite reported it green, andthe defect behind it (#305) survived a local run of 0 failures and a full CI
Windows job. It was found only by running the test outside
run-tests.Live today:
tls/003-tls-handshakereportspassed on retry attempton theWindows job of
c5a5379. What its first attempt saw is gone.Proposed
Two halves, and the first is cheap:
usleep(20000)isdelay(20)in a coroutine context; amicrotime()deadline is a boundedattempt count.
035shows both substitutions. A test that keeps the retryshould say why in a comment, so the exemption is a decision rather than an
accident.
the second attempt; the first attempt's output belongs in the artifact, and
the Windows job should probably treat a warn as a failure once the count is
low enough to allow it.
Order matters: the count has to come down before the gate can go up, or every
job goes red on the same 337 files.
Found while fixing #305.