Skip to content

run-tests retries 337 of our 497 phpt files and reports only the second attempt #313

Description

@EdmondDantes

run-tests.php retries any test it considers flaky and prints only the second
attempt's verdict, with (warn: Test passed on retry attempt) as the whole
record. A test is flaky by its judgement when its --FILE-- section calls
disk_free_space, hrtime, microtime, sleep or usleep (is_flaky,
run-tests.php:3118, reached through error_may_be_retried). The first
attempt's output is discarded, so a defect that lands on one run in two is
invisible in suite output.

337 of our 497 phpt files call one of those in their FILE section. Counted
by extracting the FILE section of each file under tests/phpt and matching the
five names — the same test is_flaky applies.

This is not hypothetical. websocket/035-recv-queue-overflow read
client saw close: NULL on 5 of 10 runs while the suite reported it green, and
the defect behind it (#305) survived a local run of 0 failures and a full CI
Windows job. It was found only by running the test outside run-tests.

Live today: tls/003-tls-handshake reports passed on retry attempt on the
Windows job of c5a5379. What its first attempt saw is gone.

Proposed

Two halves, and the first is cheap:

  1. Take the clock out of the tests that do not need it. usleep(20000) is
    delay(20) in a coroutine context; a microtime() deadline is a bounded
    attempt count. 035 shows both substitutions. A test that keeps the retry
    should say why in a comment, so the exemption is a decision rather than an
    accident.
  2. Make the retry visible. A warned test is a test whose verdict came from
    the second attempt; the first attempt's output belongs in the artifact, and
    the Windows job should probably treat a warn as a failure once the count is
    low enough to allow it.

Order matters: the count has to come down before the gate can go up, or every
job goes red on the same 337 files.

Found while fixing #305.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions